A 2030 Roadmap for Software Engineering
Open accessThe DOI matches its Crossref record; 244 of the 244 references it cites were found in a registry or an index; 16 of the 149 sources its claims cite were checked for support.
Abstract
The landscape of software engineering has dramatically changed in recent years. The impressive advances of artificial intelligence are just the latest and most disruptive innovation that has remarkably changed the software engineering research and practice. This special issue shares a roadmap to guide the software engineering community in this confused era. This roadmap is the outcome of a 2-day intensive discussion at the 2030 Software Engineering workshop. The roadmap spotlights and discusses seven main landmarks in the new software engineering landscape: artificial intelligence for software engineering, human aspects of software engineering, software security, verification and validation, sustainable software engineering, automatic programming, and quantum software engineering. This editorial summarizes the core aspects discussed in the 37 papers that comprise the seven sections of the special issue and guides the interested readers throughout the issue. This roadmap is a living body that we will refine with follow-up workshops that will update the roadmap for a series of forthcoming ACM TOSEM special issues.
Overview
CoChat Summary
TL;DR
This editorial/roadmap from the 2030 Software Engineering workshop identifies seven landmarks shaping software engineering toward 2030, centered on generative AI’s disruptive impact. It summarizes 37 papers across seven sections and proposes open research directions for AI, human aspects, sustainability, automatic programming, security, verification/validation, and quantum software engineering.
Key findings
- The roadmap spotlights seven main areas: artificial intelligence for software engineering, human aspects of software engineering, software security, verification and validation, sustainable software engineering, automatic programming, and quantum software engineering.
- The workshop was a 2-day event co-located with ACM SIGSOFT FSE on 15–16 July 2024 in Porto De Galinhas, Brazil, with 62 participants; 58 of 76 submitted papers were invited to extend, and the special issue ultimately includes 33 selected papers plus 4 editors’ papers, totaling 37 papers.
- Generative AI is described as triggering “the deepest change in the skyline of software engineering research and practice since the Internet revolution,” shifting developers toward orchestrating, validating, and curating AI-generated artifacts.
- The roadmap emphasizes challenges such as unbiased datasets and benchmarks, reliable prompts, explainability, ethics, trust in automatically generated code, security of AI-generated code, verification of evolving human-centric ecosystems, and quantum software engineering’s need for new lifecycle approaches.
Methods
- Design: editorial/roadmap produced from intensive 2-day discussions at the 2030 Software Engineering Workshop, co-located with ACM SIGSOFT FSE on 15–16 July 2024 in Porto De Galinhas, Brazil, using liberating structures such as impromptu networking, flash keynotes, 1-2-4-All, world café, fishbowl, open space technology, and conversation café.
- Data/sample: 62 workshop participants; 58 of 76 workshop submissions were invited for extension; after ACM TOSEM peer review, 33 papers were selected for the special issue, plus 4 editors’ papers, for 37 papers across seven sections.
- Analysis: qualitative synthesis organized around seven research areas; each section overviews state of the art/practice, discusses challenges and trends, and proposes a roadmap, using McLuhan’s tetrads to illustrate disruptive impacts.
Limitations
- The paper states the roadmap is “a living body” to be refined with follow-up workshops and forthcoming ACM TOSEM special issues, and that “any roadmap quickly ages.”
- The paper states that the quantum software engineering community’s limited access to quantum hardware may limit innovations in that area.
- Evident from the text: this is an editorial/roadmap synthesizing workshop discussions and 37 papers, not a primary empirical study or systematic review with a formal data-collection and analysis protocol; no quantitative evaluation of the roadmap’s outcomes is reported.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
11 Valid · 6 Not valid · 79 Not checked · 105 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“The successful use of zero-shot and one-shot learning to summarize the code [229, 235] and of conversational prompts for automatic program repair [222-224, 230] leads to new efficient prompting strategies in software engineering.”
What the sources say
- Not checked yetcites [229]
- Not checked yetcites [235]
- Not checked yetcites [222]
- Not checked yetcites [223]
- Not checked yetcites [224]
- Supportedcites [230] source
The citation to CREF is attached to the conversational-prompting part of the sentence, not the code-summarization part. Section 4.3, “Conversational Program Repair” (page unavailable), reports that CREF uses historical conversational entries and improves repair performance over the baseline. The broader conclusion about new prompting strategies is the citing authors’ synthesis. (LLM, conf=0.91)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The inadequate evaluation of deep learning systems can have significant consequences, including risks to patient well-being [165] and safety [208].”
What the sources say
- Partly supportedcites [165] sourceExperimental — accuracy not yet measured
Our copy of Richens et al. (Introduction) does state that deficiencies in associative diagnostic algorithms — the class that includes Deep Learning approaches — carry consequences for patients: 'failure to disentangle correlation from causation places strong constraints on the accuracy of associative diagnostic algorithms, sometimes resulting in sub-optimal or dangerous diagnoses', and the Discussion (Diagnostic model and datasets) flags limits of validation on EHR data (mislabeled ground truth). So the 'risk to patient well-being/safety from deficient deep learning systems' part is in the paper. What the paper does NOT say is that these risks stem from inadequate EVALUATION of deep learning systems: its argument is about inadequate inference scheme (associative vs counterfactual), and its remarks on evaluation concern the difficulty of defining ground truth, not patient harm. The serious-harm figure it does give ('one in three of these misdiagnoses resulting in serious patient harm', Introduction) is credited to Singh/Graber diagnostic-error studies (its refs 1,4), not to deep learning evaluation. (LLM, conf=0.65)
- Could not verifycites [208] sourceExperimental — accuracy not yet measured
Our copy of Wang et al. (2020) is a review of autonomous-vehicle safety; it nowhere addresses deep learning systems or their evaluation, and its safety consequences concern road users (drivers, pedestrians, cyclists), not patients. Searched: 'deep learning evaluation of neural network models' (only reference-list entries about LSTM-GAN/machine-learning control appear), 'patient well-being and safety risks in healthcare', 'insufficient testing of algorithms, training data, model validation', and 'consequences of system failure for human health' — the closest passages (3.1.1 Perception Error, 3.1.2 Decision Error, 5.2.2 Minimizing Decision Errors) discuss sensor/software/decision errors and the need for 'extensive and strict hardware and software testing' in AVs only. Since no passage speaks to the claim's point, the attribution can be neither confirmed nor contradicted on our copy. (LLM, conf=0.72)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
None of the sources cited for this claim have been checked yet.
In the paper
“Few studies propose new automated testing methods to address the diverse scenarios of deep learning models in software engineering [70, 195, 225].”
What the sources say
- Not checked yetcites [70]
- Not checked yetcites [195]
- Not checked yetcites [225]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Current studies that explore the use of artificial intelligence for software engineering research remain in their early stages [226].”
What the sources say
- Not checked yetcites [226]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Current metrics are limited in evaluating accuracy, precision, recall, and perplexity [37, 110, 111, 227].”
What the sources say
- Not checked yetcites [37]
- Not checked yetcites [110]
- Not checked yetcites [111]
- Not checked yetcites [227]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
27 more claims
Create a free account to see all 201 claims and to verify the 133 references still open.
- Probably not a substantial claim. 149
- It may state two separate claims as one. 83
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 6.1 State of the Art and Trends · p. 32
“The editors' paper [117] in this special issue provides a comprehensive review of the literature on the use of large language models for detecting and preparing vulnerabilities.”
§ 1 Introduction · p. 3
“These papers address four hot topics: Artificial Intelligence for Software Engineering [33], Software Engineering by and for Humans [1], Automatic Programming [117], and Software Security Analysis [26].”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Overview of the Editorial and the Special Issue · p. 7
“Murillo et al.'s [134] and Ramalho et al.'s [162] papers that comprise Section 8 of this special issue”
§ 8.1.1 Requirements Engineering. · p. 39
“Murillo et al.'s paper [133] in this special issue briefly mentions that quantum requirements engineering will inevitably deviate from classic requirement engineering approaches.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3.1 State of the Art and Practice · p. 18
“propose six interconnected themes:”
§ 3.1 State of the Art and Practice · p. 18
“individual capabilities, team capabilities, product, social impact, and human aspects.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4 Sustainable Software Engineering · p. 22
“the 1972 Brundtland 5 report of the UN World Commission on Environment and Development [24]”
§ References · p. 45
“[24] Gro Harlem Brundtland. 1987. Our Common Future World Commission on Environment and Developement . UN World Commission on Environment and Development.”
Citations
Cites
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682
The vision of autonomic computing
Kephart, J. O., & Chess, D. M. (2003) · Computer · doi:10.1109/mc.2003.1160055 · Cited by 6,572
Biamonte, J., Wittek, P., Pancotti, N., Rebentrost, P., Wiebe, N., & Lloyd, S. (2017) · Nature · doi:10.1038/nature23474 · Cited by 4,779
Proceedings of the 21st International Conference on Neural Information Processing Systems
Koller, D., Schuurmans, D., Bengio, Y., & Bottou, L. (2008) · Cited by 4,461
Digital Twin: Mitigating Unpredictable, Undesirable Emergent Behavior in Complex Systems
Grieves, M., & Vickers, J. (2017) · Transdisciplinary Perspectives on Complex Systems · doi:10.1007/978-3-319-38756-7_4 · Cited by 3,727
Transdisciplinary research in sustainability science: Practice, principles, and challenges
Lang, D. J., Wiek, A., Bergmann, M., Stauffacher, M., Martens, P., Moll, P., Swilling, M., & Thomas, C. J. (2012) · Sustainability Science · doi:10.1007/s11625-011-0149-x · Cited by 2,925
Guidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439
Six Transformations to achieve the Sustainable Development Goals
Sachs, J. D., Schmidt-Traub, G., Mazzucato, M., Messner, D., Nakicenovic, N., & Rockström, J. (2019) · Nature Sustainability · doi:10.1038/s41893-019-0352-9 · Cited by 2,325
From Millennium Development Goals to Sustainable Development Goals
Sachs, J. D. (2012) · The Lancet · doi:10.1016/s0140-6736(12)60685-0 · Cited by 2,269
Licklider, J. C. R. (1960) · IRE Transactions on Human Factors in Electronics · doi:10.1109/thfe2.1960.4503259 · Cited by 1,938
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888
Defects4J: a database of existing faults to enable controlled testing studies for Java programs
Just, R., Jalali, D., & Ernst, M. D. (2014) · Proceedings of the 2014 International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628055 · Cited by 1,447
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
EvoSuite: automatic test suite generation for object-oriented software
Fraser, G., & Arcuri, A. (2011) · Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering · doi:10.1145/2025113.2025179 · Cited by 1,110
The oracle problem in software testing: A survey
Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., & Yoo, S. (2015) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2014.2372785 · Cited by 1,104
Sustainable AI: AI for sustainability and the sustainability of AI
van Wynsberghe, A. (2021) · AI and Ethics · doi:10.1007/s43681-021-00043-6 · Cited by 939
A systematic evaluation of large language models of codePreprint
Xu, F. F., Alon, U., Neubig, G., & Hellendoorn, V. (2022) · MAPS@PLDI · doi:10.48550/arxiv.2202.13169 · Cited by 928
Action understanding as inverse planning
Baker, C. L., Saxe, R., & Tenenbaum, J. B. (2009) · Cognition · doi:10.1016/j.cognition.2009.07.005 · Cited by 920
Machine Learning Testing: Survey, Landscapes and Horizons
Zhang, J. M., Harman, M., Ma, L., & Liu, Y. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2962027 · Cited by 897
Alcott, B. (2005) · Ecological Economics · doi:10.1016/j.ecolecon.2005.03.020 · Cited by 859
Software Engineering for Self-Adaptive Systems: A Second Research Roadmap
de Lemos, R., Giese, H., Müller, H. A., Shaw, M., Andersson, J., Litoiu, M., Schmerl, B., Tamura, G., Villegas, N. M., Vogel, T., et al. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35813-5_1 · Cited by 716
Self-citationImproving the accuracy of medical diagnosis with causal machine learning
Richens, J. G., Lee, C. M., & Johri, S. (2020) · Nature Communications · doi:10.1038/s41467-020-17419-7 · Cited by 598
Fraser, G., & Arcuri, A. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2012.14 · Cited by 586
Software Testing With Large Language Models: Survey, Landscape, and Vision
Wang, J., Huang, Y., Chen, C., Liu, Z., Wang, S., & Wang, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3368208 · Cited by 573
A Survey on Metamorphic Testing
Segura, S., Fraser, G., Sanchez, A. B., & Ruiz-Cortes, A. (2016) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2532875 · Cited by 563
Randoop: feedback-directed random testing for Java
Pacheco, C., & Ernst, M. D. (2007) · Companion to the 22nd ACM SIGPLAN conference on Object-oriented programming systems and applications companion · doi:10.1145/1297846.1297902 · Cited by 558
Deep learning code fragments for code clone detection
White, M., Tufano, M., Vendome, C., & Poshyvanyk, D. (2016) · Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/2970276.2970326 · Cited by 558
Self-citationA systematic evaluation of large language models of code
Xu, F. F., Alon, U., Neubig, G., & Hellendoorn, V. J. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534862 · Cited by 540
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2102.04664 · Cited by 536
Metamorphic Testing: A New Approach for Generating Next Test Cases
Chen, T. Y., Cheung, S.-C., & Yiu, S.-M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.12543 · Cited by 448
Automated Program Repair in the Era of Large Pre-trained Language Models
Xia, C. S., Wei, Y., & Zhang, L. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00129 · Cited by 433
An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation
Schäfer, M., Nadi, S., Eghbali, A., & Tip, F. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3334955 · Cited by 431
GitHub Copilot AI pair programmer: Asset or Liability?
Moradi Dakhel, A., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., & Jiang, Z. M. (. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111734 · Cited by 417
GitHub Copilot AI pair programmer: Asset or Liability?
Dakhel, A. M., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., Ming, Z., & Jiang (2022) · arXiv · doi:10.48550/arxiv.2206.15331 · Cited by 417
Akata, Z., Balliet, D., de Rijke, M., Dignum, F., Dignum, V., Eiben, G., Fokkens, A., Grossi, D., Hindriks, K., Hoos, H., et al. (2020) · Computer · doi:10.1109/mc.2020.2996587 · Cited by 415
Zhou, X., Peng, X., Xie, T., Sun, J., Ji, C., Li, W., & Ding, D. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2887384 · Cited by 410
Le Goues, C., Pradel, M., & Roychoudhury, A. (2019) · Communications of the ACM · doi:10.1145/3318162 · Cited by 410
Self-citationEmpirical review of automated analysis tools on 47,587 Ethereum smart contracts
Durieux, T., Ferreira, J. F., Abreu, R., & Cruz, P. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380364 · Cited by 409
Rahman, A., Hossain, M. S., Muhammad, G., Kundu, D., Debnath, T., Rahman, M., Khan, M. S. I., Tiwari, P., & Band, S. S. (2023) · Cluster Computing · doi:10.1007/s10586-022-03658-4 · Cited by 399
Liu, J., Xia, C. S., Wang, Y., & Zhang, L. (2023) · arXiv · doi:10.48550/arxiv.2305.01210 · Cited by 390
Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code
Pan, R., Ibrahimzada, A. R., Krishna, R., Sankar, D., Wassi, L. P., Merler, M., Sobolev, B., Pavuluri, R., Sinha, S., & Jabbarvand, R. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639226 · Cited by 349
Software testing and analysis: process, principles, and techniques
(2008) · Choice Reviews Online · doi:10.5860/choice.46-0935 · Cited by 340
Self-citationWhere do we stand on the 17 Sustainable Development Goals? An overview on progress
Halkos, G., & Gkampoura, E.-C. (2021) · Economic Analysis and Policy · doi:10.1016/j.eap.2021.02.001 · Cited by 338
The ManyBugs and IntroClass benchmarks for automated repair of C programs
Le Goues, C., Holtschulte, N., Smith, E. K., Brun, Y., Devanbu, P., Forrest, S., & Weimer, W. (2015) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2015.2454513 · Cited by 326
Lessons from system dynamics modeling
Forrester, J. W. (1987) · System Dynamics Review · doi:10.1002/sdr.4260030205 · Cited by 322
Mishra, M., Desul, S., Santos, C. A. G., Mishra, S. K., Kamal, A. H. M., Goswami, S., Kalumba, A. M., Biswal, R., da Silva, R. M., dos Santos, C. A. C., et al. (2023) · Environment, Development and Sustainability · doi:10.1007/s10668-023-03225-w · Cited by 317
Automated Unit Test Improvement using Large Language Models at Meta
Alshahwan, N., Chheda, J., Finogenova, A., Gokkaya, B. U., Harman, M., Harper, I., Marginean, A., Sengupta, S., & Wang, E. (2024) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3663529.3663839 · Cited by 316
Improved Code Summarization via a Graph Neural Network
LeClair, A., Haque, S., Wu, L., & McMillan, C. (2020) · Proceedings of the 28th International Conference on Program Comprehension · doi:10.1145/3387904.3389268 · Cited by 312
The GREENSOFT Model: A reference model for green and sustainable software and its engineering
Naumann, S., Dick, M., Kern, E., & Johann, T. (2011) · Sustainable Computing: Informatics and Systems · doi:10.1016/j.suscom.2011.06.004 · Cited by 308
CodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models
Lemieux, C., Inala, J. P., Lahiri, S. K., & Sen, S. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00085 · Cited by 306
Sustainability Design and Software: The Karlskrona Manifesto
Becker, C., Chitchyan, R., Duboc, L., Easterbrook, S., Penzenstadler, B., Seyff, N., & Venters, C. C. (2015) · 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering · doi:10.1109/icse.2015.179 · Cited by 303
Self-citationPervasive label errors in test sets destabilize machine learning benchmarks
Northcutt, C. G., Athalye, A., & Mueller, J. (2021) · arXiv · doi:10.48550/arxiv.2103.14749 · Cited by 293
Toward automatic program synthesis
Manna, Z., & Waldinger, R. J. (1971) · Communications of the ACM · doi:10.1145/362566.362568 · Cited by 285
Wang, J., Zhang, L., Huang, Y., Zhao, J., & Bella, F. (2020) · Journal of Advanced Transportation · doi:10.1155/2020/8867757 · Cited by 261
Automated Repair of Programs from Large Language Models
Fan, Z., Gao, X., Mirchev, M., Roychoudhury, A., & Tan, S. H. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00128 · Cited by 256
Self-citationProceedings of the 28th Annual ACM Symposium on Applied Computing
(2013) · doi:10.1145/2480362 · Cited by 249
He, J., Treude, C., & Lo, D. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3712003 · Cited by 246
LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead
He, J., Treude, C., & Lo, D. (2024) · arXiv · doi:10.48550/arxiv.2404.04834 · Cited by 246
Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps
Moran, K., Bernal-Cardenas, C., Curcio, M., Bonett, R., & Poshyvanyk, D. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2844788 · Cited by 239
Self-citationOn the assessment of generative AI in modeling tasks: an experience report with ChatGPT and UML
Cámara, J., Troya, J., Burgueño, L., & Vallecillo, A. (2023) · Software and Systems Modeling · doi:10.1007/s10270-023-01105-5 · Cited by 234
Shamshiri, S., Just, R., Rojas, J. M., Fraser, G., McMinn, P., & Arcuri, A. (2015) · 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2015.86 · Cited by 230
On the Evaluation of Large Language Models in Unit Test Generation
Yang, L., Yang, C., Gao, S., Wang, W., Wang, B., Zhu, Q., Chu, X., Zhou, J., Liang, G., Wang, Q., et al. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695529 · Cited by 230
Framing sustainability as a property of software quality
Lago, P., Koçak, S. A., Crnkovic, I., & Penzenstadler, B. (2015) · Communications of the ACM · doi:10.1145/2714560 · Cited by 229
Self-citationA survey on deep learning for software engineering
Yang, Y., Xia, X., Lo, D., & Grundy, J. (2022) · ACM Computing Surveys · doi:10.1145/3505243 · Cited by 223
A Survey on Simulators for Testing Self-Driving Cars
Kaur, P., Taghavi, S., Tian, Z., & Shi, W. (2021) · International Conference on Connected and Autonomous Driving (MetroCAD) · doi:10.1109/metrocad51599.2021.00018 · Cited by 211
Kute, D. V., Pradhan, B., Shukla, N., & Alamri, A. (2021) · IEEE Access · doi:10.1109/access.2021.3086230 · Cited by 209
Automated Program Repair via Conversation: Fixing 162 out of 337 Bugs for $0.42 Each using ChatGPT
Xia, C. S., & Zhang, L. (2024) · Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3650212.3680323 · Cited by 203
AutoCodeRover: Autonomous Program Improvement
Zhang, Y., Ruan, H., Fan, Z., & Roychoudhury, A. (2024) · Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3650212.3680384 · Cited by 202
Self-citationAutoCodeRover: Autonomous Program Improvement
Zhang, Y., Ruan, H., Fan, Z., & Roychoudhury, A. (2024) · arXiv · doi:10.48550/arxiv.2404.05427 · Cited by 202
Self-citationSoftware sustainability: Research and practice from a software architecture viewpoint
Venters, C. C., Capilla, R., Betz, S., Penzenstadler, B., Crick, T., Crouch, S., Nakagawa, E. Y., Becker, C., & Carrillo, C. (2018) · Journal of Systems and Software · doi:10.1016/j.jss.2017.12.026 · Cited by 201
Self-citationToward Deep Learning Software Repositories
White, M., Vendome, C., Linares-Vasquez, M., & Poshyvanyk, D. (2015) · 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories · doi:10.1109/msr.2015.38 · Cited by 200
Self-citationQuantum Software Engineering: Roadmap and Challenges Ahead
Murillo, J. M., García-Alonso, J. M., Moguel, E., Barzen, J., Leymann, F., Ali, S., Yue, T., Arcaini, P., Pérez‐Castillo, R., de Guzmán, I. G., et al. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3712002 · Cited by 199
Self-citationCEUR Workshop Proceedings, Vol. 64
Euzenat, J., Gómez-Pérez, A., Guarino, N., & Stuckenschmidt, H. (2002) · Cited by 194
Requirements: The key to sustainability
Becker, C., Betz, S., Chitchyan, R., Duboc, L., Easterbrook, S. M., Penzenstadler, B., Seyff, N., & Venters, C. C. (2016) · IEEE Software · doi:10.1109/ms.2015.158 · Cited by 193
Self-citationPataranutaporn, P., Liu, R., Finn, E., & Maes, P. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00720-7 · Cited by 192
ReCode: Robustness Evaluation of Code Generation ModelsPreprint
Wang, S., Li, Z., Qian, H., Yang, C., Wang, Z., Shang, M., Kumar, V., Tan, S., Ray, B., Bhatia, P., et al. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2212.10264 · Cited by 189
Chen, C., Su, T., Meng, G., Xing, Z., & Liu, Y. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180240 · Cited by 188
Sarker, I. H., Janicke, H., Mohsin, A., Gill, A., & Maglaras, L. (2024) · ICT Express · doi:10.1016/j.icte.2024.05.007 · Cited by 188
The Rise of Artificial Intelligence under the Lens of Sustainability
Khakurel, J., Penzenstadler, B., Porras, J., Knutas, A., & Zhang, W. (2018) · Technologies · doi:10.3390/technologies6040100 · Cited by 187
Self-citationConversational Automated Program RepairPreprint
Xia, C. S., & Zhang, L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.13246 · Cited by 186
NL2Type: Inferring JavaScript Function Types from Natural Language Information
Malik, R. S., Patra, J., & Pradel, M. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2019.00045 · Cited by 169
Pohl, J., Hilty, L. M., & Finkbeiner, M. (2019) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2019.02.018 · Cited by 164
Evaluating and Improving ChatGPT for Unit Test Generation
Yuan, Z., Liu, M., Ding, S., Wang, K., Chen, Y., Peng, X., & Lou, Y. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3660783 · Cited by 163
Calero, C., & Piattini, M. G. (2015) · Springer eBooks · doi:10.1007/978-3-319-08581-4 · Cited by 159
Research Directions in Software Supply Chain Security
Williams, L. A., Benedetti, G., Hamer, S., Paramitha, R., Rahman, I., Tamanna, M., Tystahl, G., Zahan, N., Morrison, P. J., Acar, Y., et al. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3714464 · Cited by 158
Effective test generation using pre-trained Large Language Models and mutation testing
Dakhel, A. M., Nikanjam, A., Majdinasab, V., Khomh, F., & Desmarais, M. C. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107468 · Cited by 157
An Empirical Study of Model-Agnostic Techniques for Defect Prediction Models
Jiarpakdee, J., Tantithamthavorn, C. K., Dam, H. K., & Grundy, J. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.2982385 · Cited by 148
Assessing the quality of GitHub copilot’s code generation
Yetistiren, B., Ozsoy, I., & Tuzun, E. (2022) · Proceedings of the 18th International Conference on Predictive Models and Data Analytics in Software Engineering · doi:10.1145/3558489.3559072 · Cited by 143
Predicting Defective Lines Using a Model-Agnostic Technique
Wattanakriengkrai, S., Thongtanunam, P., Tantithamthavorn, C., Hata, H., & Matsumoto, K. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.3023177 · Cited by 137
A Systematic Literature Review on the Use of Deep Learning in Software Engineering Research
Watson, C., Cooper, N., Palacio, D. N., Moran, K., & Poshyvanyk, D. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3485275 · Cited by 137
Self-citationImproving ChatGPT Prompt for Code GenerationPreprint
Liu, C., Bao, X., Zhang, H., Zhang, N., Hu, H., Zhang, X., & Yan, M. L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.08360 · Cited by 135
An Empirical Study on the Usage of Transformer Models for Code Completion
Ciniselli, M., Cooper, N., Pascarella, L., Mastropaolo, A., Aghajani, E., Poshyvanyk, D., Penta, M. D., & Bavota, G. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3128234 · Cited by 134
Self-citationAutomatic Programming: Large Language Models and Beyond
Lyu, M. R.-T., Ray, B., Roychoudhury, A., Tan, S. H., & Thongtanunam, P. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708519 · Cited by 131
Self-citationLarge Language Model Supply Chain: A Research Agenda
Wang, S., Zhao, Y., Hou, X., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708531 · Cited by 125
The Current Challenges of Software Engineering in the Era of Large Language Models
Gao, C., Hu, X., Gao, S., Xia, X., & Jin, Z. G. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3712005 · Cited by 123
Explainable software analytics
Dam, H. K., Tran, T., & Ghose, A. (2018) · Proceedings of the 40th International Conference on Software Engineering: New Ideas and Emerging Results · doi:10.1145/3183399.3183424 · Cited by 120
Is ChatGPT the Ultimate Programming Assistant -- How far is it?
Tian, H., Lu, W., Li, T. O., Tang, X., Cheung, S.-C., Klein, J., & Bissyandé, T. F. (2023) · arXiv · doi:10.48550/arxiv.2304.11938 · Cited by 114
2023 IEEE Symposium on Security and Privacy (SP)
(2023) · doi:10.1109/sp46215.2023 · Cited by 111
TOGLL: Correct and Strong Test Oracle Generation with LLMS
Hossain, S. B., & Dwyer, M. B. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00098 · Cited by 110
What Makes Good In-Context Demonstrations for Code Intelligence Tasks with LLMs?
Gao, S., Wen, X.-C., Gao, C., Wang, W., Zhang, H., & Lyu, M. R. (2023) · 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00109 · Cited by 106
Pynguin: automated unit test generation for Python
Lukasczyk, S., & Fraser, G. (2022) · Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings · doi:10.1145/3510454.3516829 · Cited by 101
Using Transfer Learning for Code-Related Tasks
Mastropaolo, A., Cooper, N., Palacio, D. N., Scalabrino, S., Poshyvanyk, D., Oliveto, R., & Bavota, G. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3183297 · Cited by 98
Self-citationVenters, C. C., Lau, L., Griffiths, M. K., Holmes, V., Ward, R., Jay, C., Dibsdale, C., & Xu, J. (2014) · Journal of Open Research Software · doi:10.5334/jors.ao · Cited by 97
PyExplainer: Explaining the Predictions of Just-In-Time Defect Models
Pornprasit, C., Tantithamthavorn, C. K., Jiarpakdee, J., Fu, M. C., & Thongtanunam, P. (2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678763 · Cited by 93
Continuous Incident Triage for Large-Scale Online Service Systems
Chen, J., He, X., Lin, Q., Zhang, H., Hao, D., Gao, F., Xu, Z., Dang, Y., & Zhang, D. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00042 · Cited by 92
A Survey on Data Contamination for Large Language ModelsPreprint
Cheng, Y., Chang, Y. K., & Wu, Y. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.14425 · Cited by 90
Impediments for software test automation: A systematic literature review
Wiklund, K., Eldh, S., Sundmark, D., & Lundqvist, K. (2017) · Software Testing Verification and Reliability · doi:10.1002/stvr.1639 · Cited by 89
Suggesting natural method names to check name consistencies
Nguyen, S., Phan, H., Le, T., & Nguyen, T. N. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380926 · Cited by 89
Duboc, L., Penzenstadler, B., Porras, J., Akinli Kocak, S., Betz, S., Chitchyan, R., Leifler, O., Seyff, N., & Venters, C. C. (2020) · Requirements Engineering · doi:10.1007/s00766-020-00336-y · Cited by 88
Self-citationAn empirical study of log analysis at Microsoft
He, S., Zhang, X., He, P., Xu, Y. D., Li, L., Kang, Y., Ma, M., Wei, Y., Dang, Y., Rajmohan, S., et al. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3558963 · Cited by 86
Towards a definition of sustainability in and for software engineering
Penzenstadler, B. (2013) · Proceedings of the 28th Annual ACM Symposium on Applied Computing · doi:10.1145/2480362.2480585 · Cited by 82
Self-citationSoftware Engineering by and for Humans in an AI Era
Abrahão, S., Grundy, J., Pezzè, M., Storey, M., & Tamburri, D. A. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715111 · Cited by 81
Self-citationWhen software engineering meets quantum computing
Ali, S., Yue, T., & Abreu, R. (2022) · Communications of the ACM · doi:10.1145/3512340 · Cited by 80
Self-citationSustainability in computing education: A systematic literature review
Peters, A.-K., Capilla, R., Coroamă, V. C., Heldal, R., Lago, P., Leifler, O., Moreira, A., Fernandes, J. P., Penzenstadler, B., Porras, J., et al. (2024) · ACM Transactions on Computing Education · doi:10.1145/3639060 · Cited by 78
Self-citationHuang, H., Bu, X., Zhou, H., Qu, Y., Liu, J., Yang, M., Xu, B., & Zhao, T. (2024) · arXiv · doi:10.48550/arxiv.2403.02839 · Cited by 78
Test Oracle Automation in the Era of LLMs
Molina, F., Gorla, A., & d’Amorim, M. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715107 · Cited by 73
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2310.06770 · Cited by 72
Testing and Debugging Quantum Programs: The Road to 2030
Ramalho, N. C. L., de Souza, H. A., & Chaim, M. L. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715106 · Cited by 69
DeceptPrompt: Exploiting LLM-driven Code Generation via Adversarial Natural Language InstructionsPreprint
Wu, F., Liu, X., & Xiao, C. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.04730 · Cited by 69
Guldner, A., Bender, R., Calero, C., Fernando, G. S., Funke, M., Gröger, J., Hilty, L. M., Hörnschemeyer, J., Hoffmann, G.-D., Junger, D., et al. (2024) · Future Generation Computer Systems · doi:10.1016/j.future.2024.01.033 · Cited by 67
Deep Learning or Classical Machine Learning? An Empirical Study on Log-Based Anomaly Detection
Yu, B., Yao, J., Fu, Q., Zhong, Z., Xie, H., Wu, Y., Ma, Y., & He, P. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3623308 · Cited by 65
Enhancing Automated Requirements Traceability by Resolving Polysemy
Wang, W., Niu, N., Liu, H., & Niu, Z. (2018) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re.2018.00-53 · Cited by 63
A co-design framework of neural networks and quantum circuits towards quantum advantage
Jiang, W., Xiong, J., & Shi, Y. (2021) · Nature Communications · doi:10.1038/s41467-020-20729-5 · Cited by 63
The future of AI-driven software engineering
Terragni, V., Vella, A., Roop, P., & Blincoe, K. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715003 · Cited by 62
Sustainable software engineering: Reflections on advances in research and practice
Venters, C. C., Capilla, R., Nakagawa, E. Y., Betz, S., Penzenstadler, B., Crick, T., & Brooks, I. (2023) · Information and Software Technology · doi:10.1016/j.infsof.2023.107316 · Cited by 59
Self-citationA Catalogue Supporting Software Sustainability Design
Oyedeji, S., Seffah, A., & Penzenstadler, B. (2018) · Sustainability · doi:10.3390/su10072296 · Cited by 58
Self-citationInsolvent: How to Reorient Computing for Just Sustainability
Becker, C. (2023) · doi:10.7551/mitpress/14668.001.0001 · Cited by 56
Bhatia, S., Gandhi, T., Kumar, D., & Jalote, P. (2024) · Proceedings of the 1st International Workshop on Large Language Models for Code · doi:10.1145/3643795.3648396 · Cited by 55
LLM App Store Analysis: A Vision and Roadmap
Zhao, Y., Hou, X., Wang, S., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708530 · Cited by 55
Software engineering for sustainability: Find the leverage points!
Penzenstadler, B., Duboc, L., Venters, C. C., Betz, S., Seyff, N., Wnuk, K., Chitchyan, R., Easterbrook, S. M., & Becker, C. (2018) · IEEE Software · doi:10.1109/ms.2018.110154908 · Cited by 53
Self-citationArtificial Intelligence for Software Engineering: The Journey So Far and the Road Ahead
Ahmed, I., Aleti, A., Cai, H., Chatzigeorgiou, A., He, P., Hu, X., Pezzè, M., Poshyvanyk, D., & Xia, X. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3719006 · Cited by 51
Self-citationElhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., Hatfield-Dodds, Z., Lasenby, R., Drain, D., Chen, C., et al. (2022) · arXiv · doi:10.48550/arxiv.2209.10652 · Cited by 51
A Framework for Sustainability
Tainter, J. A. (2003) · World Futures · doi:10.1080/02604020310132 · Cited by 50
From Today’s Code to Tomorrow’s Symphony: The AI Transformation of Developer’s Routine by 2030
Qiu, K., Puccinelli, N., Ciniselli, M., & Grazia, L. D. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3709353 · Cited by 50
Extending MAPE-K to support human-machine teaming
Cleland‐Huang, J., Agrawal, A., Vierhauser, M., Murphy, M. N., & Prieto, M. (2022) · International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS) · doi:10.1145/3524844.3528054 · Cited by 49
Impact of log parsing on deep learning-based anomaly detection
Khan, Z. A., Shin, D., Bianculli, D., & Briand, L. C. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10533-w · Cited by 49
Characterising Sustainability Requirements: A New Species Red Herring or Just an Odd Fish?
Venters, C. C., Seyff, N., Becker, C., Betz, S., Chitchyan, R., Duboc, L., Mcintyre, D., & Penzenstadler, B. (2017) · 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Society Track (ICSE-SEIS) · doi:10.1109/icse-seis.2017.2 · Cited by 44
Self-citationCode to comment "translation": data, metrics, baselining & evaluation
Gros, D., Sezhiyan, H., Devanbu, P., & Yu, Z. (2020) · Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3324884.3416546 · Cited by 43
Agile Methodology for the Standardization of Engineering Requirements Using Large Language Models
Ray, A. T., Cole, B. F., Fischer, O. J. P., Bhat, A. P., White, R. T., & Mavris, D. N. (2023) · Systems · doi:10.3390/systems11070352 · Cited by 43
Studying the explanations for the automated prediction of bug and non-bug issues using LIME and SHAP
Schulte, L., Ledel, B., & Herbold, S. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10469-1 · Cited by 43
Shi, J., Yang, Z., & Lo, D. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708525 · Cited by 43
Call Me Maybe: Using NLP to Automatically Generate Unit Test Cases Respecting Temporal Constraints
Blasi, A., Gorla, A., Ernst, M. D., & Pezzè, M. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556961 · Cited by 42
Self-citationReCode: Robustness Evaluation of Code Generation Models
Wang, S., Li, Z., Qian, H., Yang, C., Wang, Z., Shang, M., Kumar, V., Tan, S., Ray, B., Bhatia, P., et al. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.773 · Cited by 42
Evaluating large language models for software testing
Li, Y., Liu, P., Wang, H., Chu, J., & Wong, W. E. (2025) · Computer Standards & Interfaces · doi:10.1016/j.csi.2024.103942 · Cited by 42
Birchler, C., Khatiri, S., Rani, P., Kehrer, T., & Panichella, S. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3711906 · Cited by 42
Who is the advocate? Stakeholders for sustainability
Penzenstadler, B., Femmer, H., & Richardson, D. J. (2013) · IEEE/ACM International Workshop on Green and Sustainable Software (GREENS) · doi:10.1109/greens.2013.6606424 · Cited by 41
Self-citationCREF: An LLM-Based Conversational Software Repair Framework for Programming Tutors
Yang, B., Tian, H., Pian, W., Yu, H., Wang, H., Klein, J., Bissyandé, T. F., & Jin, S. (2024) · Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3650212.3680328 · Cited by 41
Teach sustainability in software engineering?
Penzenstadler, B., & Fleischmann, A. (2011) · 2011 24th IEEE-CS Conference on Software Engineering Education and Training (CSEE&T) · doi:10.1109/cseet.2011.5876124 · Cited by 40
Self-citationQuantum computers: what are they good for?
Brooks, M. (2023) · Nature · doi:10.1038/d41586-023-01692-9 · Cited by 40
LLM for Mobile: An Initial Roadmap
Chen, D., Liu, Y., Zhou, M., Zhao, Y., Wang, H., Wang, S., Chen, X., Bissyandé, T. F., Klein, J., & Li, L. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708528 · Cited by 39
Learning Code-Query Interaction for Enhancing Code Searches
Li, W., Qin, H., Yan, S., Shen, B., & Chen, Y. (2020) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme46990.2020.00021 · Cited by 37
A Survey of Field-based Testing Techniques
Bertolino, A., Braione, P., Angelis, G. D., Gazzola, L., Kifetew, F., Mariani, L., Orrù, M., Pezzè, M., Pietrantuono, R., Russo, S., et al. (2022) · ACM Computing Surveys · doi:10.1145/3447240 · Cited by 37
Self-citationCan we find bugs using LLM-generated oracles?Preprint
Konstantinou, M., Degiovanni, R., & Papadakis, M. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.21136 · Cited by 37
Contemporary Software Modernization: Strategies, Driving Forces, and Research Opportunities
Assunção, W. K. G., Marchezan, L., Arkoh, L., Egyed, A. F., & Ramler, R. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708527 · Cited by 37
Feature Maps: A Comprehensible Software Representation for Design Pattern Detection
Thaller, H., Linsbauer, L., & Egyed, A. (2019) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2019.8667978 · Cited by 37
Large Language Models: The Next Frontier for Variable Discovery within Metamorphic Testing?
Tsigkanos, C., Rani, P., Müller, S., & Kehrer, T. (2023) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner56733.2023.00070 · Cited by 35
Engineering Digital Systems for Humanity: A Research Roadmap
Autili, M., Sanctis, M. D., Inverardi, P., & Pelliccione, P. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3712006 · Cited by 35
Generative AI in Software Engineering Must Be Human-Centered: The Copenhagen Manifesto
Russo, D., Baltes, S., van Berkel, N., Avgeriou, P., Calefato, F., Cabrero-Daniel, B., Catolino, G., Cito, J., Ernst, N., Fritz, T., et al. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112115 · Cited by 34
Self-citationSoftware Fairness Debt: Building a Research Agenda for Addressing Bias in AI Systems
Santos, R. D. S., Fronchetti, F., Freire, S., & Spinola, R. O. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3709357 · Cited by 34
Automation in Model-Driven Engineering: A Look Back, and Ahead
Burgueño, L., Ruscio, D. D., Sahraoui, H. A., & Wimmer, M. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3712008 · Cited by 34
Deep Learning & Software Engineering: State of Research and Future DirectionsPreprint
Dévanbu, P., Dwyer, M. B., Elbaum, S. G., Lowry, M., Moran, K., Poshyvanyk, D., Ray, B., Singh, R., & Zhang, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.08525 · Cited by 34
Self-citationStudying the difference between natural and programming language corpora
Casalnuovo, C., Sagae, K., & Devanbu, P. (2019) · Empirical Software Engineering · doi:10.1007/s10664-018-9669-7 · Cited by 33
Adaptive optimization federated learning enabled digital twins in industrial IoT
Yang, W., Yang, Y. W., Xiang, W., Yu, K., Yu, K., Alonso, Á. H., Ureña, J., & Pang, Z. (2024) · Journal of Industrial Information Integration · doi:10.1016/j.jii.2024.100645 · Cited by 33
LLMs and Prompting for Unit Test Generation: A Large-Scale Evaluation
Ouédraogo, W. C., Kaboré, A. K., Tian, H., Song, Y., Koyuncu, A., Klein, J., Lo, D., & Bissyandé, T. F. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695330 · Cited by 33
Metamorphic Relation Generation: State of the Art and Research Directions
Li, R., Liu, H., Poon, P., Towey, D., Sun, C., Zheng, Z., Zhou, Z. Q., & Chen, T. Y. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708521 · Cited by 33
Open Source AI-based SE Tools: Opportunities and Challenges of Collaborative Software Learning
Lin, Z., Ma, W., Lin, T., Zheng, Y., Ge, J., Wang, J., Klein, J., Bissyandé, T. F., Liu, Y., & Li, L. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708529 · Cited by 33
Towards Generating Executable Metamorphic Relations Using Large Language Models
Shin, S. Y., Pastore, F., Bianculli, D., & Băicoianu, A. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-70245-7_9 · Cited by 32
From Triumph to Uncertainty: The Journey of Software Engineering in the AI Era
Mastropaolo, A., Escobar‐Velásquez, C., & Linares‐Vásquez, M. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3709360 · Cited by 30
Hyrynsalmi, S. M., Baltes, S., Brown, C., Prikladnicki, R., Rodríguez-Pérez, G., Serebrenik, A., Simmonds, J., Trinkenreich, B., Wang, Y., & Liebel, G. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3711904 · Cited by 30
A Prompt Learning Framework for Source Code SummarizationPreprint
Xu, T., Miao, Y., Fang, C., Qian, H., Feng, X., Chen, Z., Wang, C., Zhang, J., Sun, W., Chen, Z., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.16066 · Cited by 30
Traffic Scenarios for Automated Vehicle Testing: A Review of Description Languages and Systems
Ma, J., Che, X., Li, Y., & Lai, E. M.-K. (2021) · Machines · doi:10.3390/machines9120342 · Cited by 28
Toward model-driven sustainability evaluation
Kienzle, J., Mussbacher, G., Combemale, B., Bastin, L., Bencomo, N., Bruel, J., Becker, C., Betz, S., Chitchyan, R., Cheng, B. H. C., et al. (2020) · Communications of the ACM · doi:10.1145/3371906 · Cited by 27
Self-citationA Roadmap for Software Testing in Open-Collaborative and AI-Powered Era
Wang, Q., Wang, J., Li, M., Wang, Y., & Liu, Z. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3709355 · Cited by 26
Sustainable Software Engineering: Concepts, Challenges, and Vision
König, C., Lang, D. J., & Schaefer, I. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3709352 · Cited by 25
Saputri, T. R. D., & Lee, S. W. (2020) · Journal of Software Evolution and Process · doi:10.1002/smr.2254 · Cited by 24
Evaluating and Explaining Large Language Models for Code Using Syntactic StructuresPreprint
Palacio, D. N., Velasco, A., Rodríguez-Cárdenas, D., Moran, K., & Poshyvanyk, D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.03873 · Cited by 24
Self-citationThe Impact of Generative AI on Creativity in Software Development: A Research Agenda
Jackson, V., Vasilescu, B., Russo, D., Ralph, P., Prikladnicki, R., Izadi, M., D’Angelo, S., Inman, S., Andrade, A., & van der Hoek, A. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708523 · Cited by 23
Probing Pretrained Models of Source Codes
Troshin, S., & Chirkova, N. (2022) · Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/2022.blackboxnlp-1.31 · Cited by 23
Betz, S., Penzenstadler, B., Duboc, L., Chitchyan, R., Kocak, S. A., Brooks, I., Oyedeji, S., Porras, J., Seyff, N., & Venters, C. C. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3649597 · Cited by 23
Self-citationA Roadmap for Integrating Sustainability into Software Engineering Education
Moreira, A., Lago, P., Heldal, R., Betz, S., Brooks, I., Capilla, R., Coroamă, V. C., Duboc, L., Fernandes, J. P., Leifler, O., et al. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708526 · Cited by 22
Self-citationHow Propense Are Large Language Models at Producing Code Smells? A Benchmarking Study
Velasco, A., Rodríguez-Cárdenas, D., Alif, L. R., Palacio, D. N., & Poshyvanyk, D. (2025) · IEEE/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) · doi:10.1109/icse-nier66352.2025.00025 · Cited by 22
Self-citationCode Localization in Programming Screencasts
Alahmadi, M. D., Khormi, A., Parajuli, B., Hassel, J., Haiduc, S., & Kumar, P. (2020) · Empirical Software Engineering · doi:10.1007/s10664-019-09759-w · Cited by 21
Software Security Analysis in 2030 and Beyond: A Research Roadmap
Böhme, M., Bodden, E., Bultan, T., Cadar, C., Liu, Y., & Scanniello, G. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708533 · Cited by 21
Efficient customer incident triage via linking with system incidents
Gu, J., Wen, J., Wang, Z., Zhao, P., Luo, C. W., Kang, Y., Zhou, Y., Li, Y., Sun, J. C., Xu, Z., et al. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3417061 · Cited by 20
More sustainable artificial intelligence systems through stakeholder involvement?
Kunkel, S., Schmelzle, F., Niehoff, S., & Beier, G. (2023) · GAIA - Ecological Perspectives for Science and Society · doi:10.14512/gaia.32.s1.10 · Cited by 20
SBFT Tool Competition 2023 - Java Test Case Generation Track
Jahangirova, G., & Terragni, V. (2023) · doi:10.1109/sbft59156.2023.00025 · Cited by 19
Towards More Trustworthy and Interpretable LLMs for Code through Syntax-Grounded ExplanationsPreprint
Palacio, D. N., Rodríguez-Cárdenas, D., Velasco, A., Khati, D., Moran, K., & Poshyvanyk, D. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.08983 · Cited by 19
Self-citationToward a Theory of Causation for Interpreting Neural Code Models
Nader Palacio, D., Velasco, A., Cooper, N., Rodriguez, A., Moran, K., & Poshyvanyk, D. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3379943 · Cited by 18
Self-citationMultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation
Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps-Costin, L., Pinckney, D., Yee, M.-H., Zi, Y., Anderson, C. J., Feldman, M. Q., et al. (2022) · arXiv · doi:10.48550/arxiv.2208.08227 · Cited by 17
Prototyping Automotive Smart Ecosystems
Cioroaica, E., Kuhn, T. S., & Bauer, T. (2018) · IEEE/IFIP International Conference on Dependable Systems and Networks Workshops (DSN-W) · doi:10.1109/dsn-w.2018.00072 · Cited by 16
Automated Extraction of Requirement Entities by Leveraging LSTM-CRF and Transfer Learning
Li, M., Yang, Y., Shi, L., Wang, Q., Hu, J., Peng, X., Liao, W., & Pi, G. (2020) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme46990.2020.00029 · Cited by 15
Software Sustainability: Beyond the Tower of Babel
Venters, C. C., Kocak, S. A., Betz, S., Brooks, I., Capilla, R., Chitchyan, R., Duboc, L., Heldal, R., Moreira, A., Oyedeji, S., et al. (2021) · 2021 IEEE/ACM International Workshop on Body of Knowledge for Software Sustainability (BoKSS) · doi:10.1109/bokss52540.2021.00009 · Cited by 15
Self-citationSun, Y., Shen, X., & Zhang, K. Z. K. (2023) · Data and Information Management · doi:10.1016/j.dim.2023.100048 · Cited by 15
With Great Power Comes Great Responsibility: The Role of Software Engineers
Betz, S., & Penzenstadler, B. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715112 · Cited by 15
Self-citationFoundation Model Engineering: Engineering Foundation Models Just as Engineering Software
Ran, D., Wu, M. S., Yang, W. M., & Xie, T. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3719005 · Cited by 15
Rationales for Sequential Predictions
Vafa, K., Deng, Y., Blei, D., & Rush, A. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.807 · Cited by 14
Benchmarking Causal Study to Interpret Large Language Models for Source Code
Rodriguez-Cardenas, D., Palacio, D. N., Khati, D., Burke, H., & Poshyvanyk, D. (2023) · 2023 IEEE International Conference on Software Maintenance and Evolution (ICSME) · doi:10.1109/icsme58846.2023.00040 · Cited by 14
Self-citationSoftware Engineering for Collective Cyber-Physical Ecosystems
Casadei, R., Aguzzi, G., Audrito, G., Damiani, F., Pianini, D., Scarso, G., Torta, G., & Viroli, M. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3712004 · Cited by 12
Research works printed but not linked: 23
- Gro Harlem Brundtland. 1987. Our Common Future World Commission on Environment and Developement. UN World Commission on Environment and Development.
- Johan Cederbladh, Antonio Cicchetti, and Robbert Jongeling. n.d. A road-map to readily available early validation & verification of system behaviour in model-based systems engineering using software engineering best practices. ACM Transaction on Software Engineering and Methodology 2030 Roadmap Special Issue, 1049-331X.
- Xinyun Chen, Chang Liu, and Dawn Song.2018. Execution-guided neural program synthesis. In Proceedings of the International Conference on Learning Representations.
- Eliot Zackrone, Beiza Eken, Michal Ernst, Mauro Pezzè, Davide Molinelli, and Alberto Martin-Lopez. 2025. Tratto: A neuro-symbolic approach to deriving axiomatic test oracles. In Proceedings of the ACM ISSTA International Symposium on Software Testing and Analysis.
- Elizabeth Dinella, Gabriel Ryan, Todd Mytkowicz, and Shuvendu Lahiri. 2022. TOGA: A neural method for test oracle generation. In Proceedings of the International Conference on Software Engineering (ICSE ’22). ACM. Retrieved from https://www.microsoft.com/en-us/research/publication/toga-a-neural-method-for-test-oracle-generation/
- Guoxiang Guo Aldeida Aleti Neelofar Neelofar and Chakkrit Tantithamthavorn. 2024. MORTAR: Metamorphic multi-turn testing for LLM-based dialogue systems. arXiv:2412.15557. Retrieved from https://arxiv.org/abs/2412.15557 · arXiv
- Ashish Hooda, Mihai Christodorescu, Miltiadis Allamanis, Aaron Wilson, Kassem Fawaz, and Somesh Jha. 2024. Do large code models understand programming concepts? Counterfactual analysis for code predicates. In Proceedings of the 41st International Conference on Machine Learning (ICML’24). Article 753, 11 pages.
- Sonja Hyrynsalmi, Sebastian Baltes, Chris Brown, Rafael Prikladnicki, Perez Gema Rodriguez, Alexander Serebrenik, Jocelyn Simmonds, Bianca Trinkenreich, and Yi Wang. 2025. Bridging gaps, building futures: Advancing software developer diversity and inclusion through future-oriented research. ACM Transaction on Software Engineering and Methodology 2030 Roadmap Special Issue, 1049-331X.
- Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. 2020. Learning and evaluating contextual embedding of source code. In Proceedings of the International Conference on Machine Learning. PMLR, 5110–5121.
- Fatemeh Khayashi Behnaz Jamasb Reza Akbari and Pirooz Shamsinejadbabaki. 2022. Deep learning methods for software requirement classification: A performance study on the PURE dataset. arXiv:2211.05286. Retrieved from https://arxiv.org/abs/2211.05286 · arXiv
- Jian Li Yue Wang Michael R. Lyu and Irwin King. 2017. Code completion with neural attention and pointer networks. arXiv:1711.09573. Retrieved from https://arxiv.org/abs/1711.09573 · arXiv
- Ruijie Meng, Martin Mirchev, Marcel Böhme, and Abhik Roychoudhury. 2024. Large language model guided protocol fuzzing. In Proceedings of the 31st Annual Network and Distributed System Security Symposium (NDSS).
- Juan Manuel Murillo, Jose Garcia-Alonso, Enrique Moguel, Johanna Barzen, Frank Leymann, Shaukat Ali, Tao Yue, Paolo Arcaini, Ricardo Perez-Castillo, Ignacio García-Rodríguez de Guzmán, et al. 2025. Quantum software engineering: Roadmap and challenges ahead. ACM Transaction on Software Engineering and Methodology 2030 Roadmap Special Issue, 1049-331X.
- Birgit Penzenstadler. 2014. Infusing green: Requirements engineering for green In and through software systems. In Proceedings of the International Workshop on Requirements Engineering for Sustainable Systems (RE4SuSy@ RE), 44–53.
- Jari Porras, Colin C. Venters, Birgit Penzenstadler, Leticia Duboc, Stefanie Betz, Norbert Seyff, Saeid Heshmatisafa, and Shola Oyedeji. 2021. How could we have known? Anticipating sustainability effects of a software product. In Proceedings of the 12th International Conference Software Business (ICSOB ’21). Springer, 10–17.
- J. Schneider-Hufschmidt, T. Kühme, and U. Malinowski. 1993. Adaptive User Interfaces: Principles and Practice. North Holland, London.
- Norbert Seyff, Stefanie Betz, Leticia Duboc, Colin Venters, Christoph Becker, Ruzanna Chitchyan, Birgit Penzenstadler, and Markus Nöbauer. 2018. Tailoring requirements negotiation to sustainability. In Proceedings of the 2018 IEEE 26th International Requirements Engineering Conference (RE). IEEE, 304–314.
- Trevor Stalnaker Nathan Wintersgill Oscar Chaparro Laura A. Heymann Massimiliano Di Penta Daniel M. German and Denys Poshyvanyk. 2024. Developer perspectives on licensing and copyright issues arising from generative AI for coding. arXiv:2411.10877. Retrieved from https://arxiv.org/abs/2411.10877 · arXiv
- Colin C. Venters, Caroline Jay, L. M. S. Lau, Michael K. Griffiths, Violeta Holmes, Rupert R. Ward, Jim Austin, Charlie E. Dibsdale, and Jie Xu. 2014. Software sustainability: The modern tower of Babel. In Proceedings of the CEUR Workshop 1216 (2014), 7–12.
- Mingxuan Xiao Yan Xiao Shunhui Ji Yunhe Li Lei Xue and Pengcheng Zhang. 2025. ABFS: Natural robustness testing for LLM-based NLP software. arXiv:2503.01319. Retrieved from https://arxiv.org/abs/2503.01319 · arXiv
- Junjielong Xu, Qinan Zhang, Zhiqing Zhong, Shilin He, Chaoyun Zhang, Qingwei Lin, Dan Pei, Pinjia He, Dongmei Zhang, and Qi Zhang. 2025. OpenRCA: Can large language models locate the root cause of software failures? In Proceedings of the 13th International Conference on Learning Representations (ICLR).
- Tao Yue, Wolfgang Mauerer, Shaukat Ali, and Davide Taibi. 2023. Challenges and opportunities in quantum software architecture. Software Architecture: Research Roadmaps from the Community, 1–23.
- 10.1109/tse.2018.2887384 · DOI · Paper page
Cited by
Amalfitano, D., Metzger, A., Autili, M., Fulcini, T., Hey, T., Keim, J., Pelliccione, P., Scotti, V., Koziolek, A., Mirandola, R., et al. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3788879 · Cited by 30
Rani, P. J. E., Gopinath, K., Deepika, G., Priyadharshini, K., Dhamodaran, S., & Mohanraj, B. (2025) · International Conference on Sustainable Communication Networks and Application (ICSCNA) · doi:10.1109/icscn67106.2025.11308225 · Cited by 6
Do LLMs Generate Useful Test Oracles? An Empirical Study with an Unbiased Dataset
Molinelli, D., Grazia, L. D., Martin-Lopez, A., Ernst, M. D., & Pezzè, M. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00031 · Cited by 16
Self-citationSantos, S. D. O., Alves, R. D. M., Oliveira, F., & Araujo, F. I. S. (2025) · Brazilian Symposium on Multimedia and the Web · doi:10.5753/webmedia.2025.16160 · Cited by 4
Migliarini, P., Autili, M., Inverardi, P., & Pelliccione, P. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3801980 · Cited by 1
Metadata
- Authors
- Pezzè, M., Abrahão, S., Penzenstadler, B., Poshyvanyk, D., Roychoudhury, A., & Yue, T.
- Affiliations
- Constructor Institute (CH)
- Universitat Politècnica de València (ES)
- Lappeenranta-Lahti University of Technology (FI)
- William & Mary (US)
- National University of Singapore (SG)
- Beihang University (CN)
- Venue
- ACM Transactions on Software Engineering and Methodology
- Published
- 2025-05-26
- Volume
- 34
- Issue
- 5
- Pages
- 1-55
- ISSN
- 1049-331X (ISSN-L)
- 1557-7392
- OpenAlex
- W4409797578
- Semantic Scholar
- 14ecf35dcc975a5f5ed5e0cbaf38cc7bbfe4fc54
- Cited by
- 13
- References
- 196
- Citations by year
- 2026: 10 · 2025: 3
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Transactions on Software Engineering and Methodology (publishedVersion, cc-by)
- Chalmers Research (Chalmers University of Technology) (submittedVersion)
- Funders
- Swiss National Foundation (SNF 200021_215487)
- State Research Agency (PID2022-140106NB-I00)
- Generalitat Valenciana (CIAICO/2021/303 (AKILA))
- Denys Poshyvanyk’s research (NSF CCF-234635, CCF-2311469, CNS-2132281, and CCF-1955853)
- Singapore Ministry of Education (MoE) Tier3 (MOE-MOET32021-0001)
- State Key Laboratory of Complex & Critical Software Environment (CCSE-2024ZX-01)
- Fundamental Research Funds for the Central Universities
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Advanced Software Engineering Methodologies
- Keywords
- human aspects of software engineering, software security, verification and validation, sustainable software development, program synthesis, quantum software engineering, future of software engineering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.