An Empirical Study on Using Large Language Models to Analyze Software Supply Chain Security Failures
Open accessAbstract
As we increasingly depend on software systems, the consequences of breaches in the software supply chain become more severe. High-profile cyber attacks like SolarWinds and ShadowHammer have resulted in significant financial and data losses, underlining the need for stronger cybersecurity. One way to prevent future breaches is by studying past failures. However, traditional methods of analyzing past failures require manually reading and summarizing reports about them. Automated support could reduce costs and allow analysis of more failures. Natural Language Processing (NLP) techniques such as Large Language Models (LLMs) could be leveraged to assist the analysis of failures. In this study, we assessed the ability of Large Language Models (LLMs) to analyze historical software supply chain breaches. We used LLMs to replicate the manual analysis of 69 software supply chain security failures performed by members of the Cloud Native Computing Foundation (CNCF). We developed prompts for LLMs to categorize these by four dimensions: type of compromise, intent, nature, and impact. GPT 3.5's categorizations had an average accuracy of 68% and Bard's had an accuracy of 58% over these dimensions. We report that LLMs effectively characterize software supply chain failures when the source articles are detailed enough for consensus among manual analysts, but cannot yet replace human analysts. Future work can improve LLM performance in this context, and study a broader range of articles and failures.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs show good potential for automatically analyzing and summarizing software supply chain security vulnerabilities, especially when the data is comprehensive, but still cannot replace human analysts in this specific field.
“Their results show that LLMs show good potential, especially when the data is comprehensive, but still cannot replace human analysts in this specific field.”
From When LLMs meet cybersecurity: a systematic literature review (2025) · cites this paper as Singla et al. 2023
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398 · see citing works
Basic concepts and taxonomy of dependable and secure computing
Avižienis, A., Laprie, J.-C., Randell, B., & Landwehr, C. E. (2004) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2004.2 · Cited by 5,230 · see citing works
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · cited 1× · see citing works
“We prompted LLMs in the order of Table 3.”
MethodLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · cited 1× · see citing works
“We prompted LLMs in the order of Table 3.”
MethodA Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093 · see citing works
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.18223 · Cited by 2,090
Safeware: System Safety and Computers
Leveson, N. G. (1995) · CERN Document Server (European Organization for Nuclear Research) · Cited by 1,660 · see citing works
Educational and Psychological Measurement
(1982) · Serials Review · doi:10.1080/00987913.1982.10763392 · Cited by 1,410 · see citing works
McNutt, M. K. (2017) · Human Rights Documents online · doi:10.1163/2210-7975_hrd-9978-20190011 · Cited by 1,183 · see citing works
How can i improve my app? Classifying user reviews for software maintenance and evolution
Panichella, S., Sorbo, A. D., Guzmán, E., Visaggio, C. A., Canfora, G., & Gall, H. C. (2015) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsm.2015.7332474 · Cited by 472 · see citing works
Natural Language Processing for Requirements Engineering
Zhao, L., Alhoshan, W., Ferrari, A., Letsholo, K. J., Ajagbe, M. A., Chioasca, E.-V., & Batista-Navarro, R. (2021) · ACM Computing Surveys · doi:10.1145/3444689 · Cited by 364 · see citing works
Nikiforakis, N., Invernizzi, L., Kapravelos, A., Acker, S. V., Joosen, W., Kruegel, C., Piessens, F., & Vigna, G. (2012) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/2382196.2382274 · Cited by 279 · see citing works
Backstabber’s Knife Collection: A Review of Open Source Software Supply Chain Attacks
Ohm, M., Plate, H., Sykosch, A., & Meier, M. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-52683-2_2 · Cited by 259 · see citing works
Petroski, H. J. (1994) · Cambridge University Press eBooks · doi:10.1017/cbo9780511805073 · Cited by 252 · see citing works
SoK: Taxonomy of Attacks on Open-Source Software Supply Chains
Ladisa, P., Plate, H., Martinez, M., & Barais, O. (2023) · 2023 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp46215.2023.10179304 · Cited by 208 · see citing works
A defined process for project post mortem review
Collier, B., DeMarco, T., & Fearey, P. (1996) · IEEE Software · doi:10.1109/52.526833 · Cited by 174 · see citing works
Small World with High Risks: A Study of Security Threats in the npm EcosystemPreprint
Zimmermann, M. N., Staicu, C.-A., Tenny, C., & Pradel, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.09217 · Cited by 147 · see citing works
Detection, assessment and mitigation of vulnerabilities in open source dependencies
Ponta, S. E., Plate, H., & Sabetta, A. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09830-x · Cited by 145 · see citing works
The Experience Factory and its relationship to other Improvement Paradigms
Basili, V. R. (1993) · Lecture notes in computer science · doi:10.1007/3-540-57209-0_6 · Cited by 116 · see citing works
CyBERT: Contextualized Embeddings for the Cybersecurity Domain
Ranade, P., Piplai, A., Joshi, A., & Finin, T. (2021) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata52589.2021.9671824 · Cited by 115 · see citing works
Vulnerable open source dependencies
Николаевич, П. И., Plate, H., Ponta, S. E., Sabetta, A., & Massacci, F. (2018) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3239235.3268920 · Cited by 110 · see citing works
Automated risk identification using NLP in cloud based development environments
Vijayakumar, K., & Arun, C. (2017) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-017-0503-7 · Cited by 103 · see citing works
Towards intelligent incident management: why we need it and how we make it
Chen, Z., Kang, Y., Li, L., Zhang, X., Zhang, H., Xu, H., Zhou, Y., Yang, L., Sun, J., Xu, Z., et al. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3417055 · Cited by 99 · see citing works
Time pressure in software engineering: A systematic review
Kuutila, M., Mäntylä, M. V., Farooq, U., & Claes, M. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2020.106257 · Cited by 98 · see citing works
Design and code inspections to reduce errors in program development
Fagan, M. E. (1999) · IBM Systems Journal · doi:10.1147/sj.382.0258 · Cited by 97 · see citing works
NLP-assisted software testing: A systematic mapping of the literature
Garousi, V., Bauer, S., & Felderer, M. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106321 · Cited by 81 · see citing works
A Systematic Analysis of the Juniper Dual EC Incident
Checkoway, S., Maskiewicz, J., Garman, C., Fried, J., Cohney, S., Green, M., Heninger, N., Weinmann, R.-P., Rescorla, E. K., & Shacham, H. (2016) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/2976749.2978395 · Cited by 74 · see citing works
An Empirical Study on the Correctness of Formally Verified Distributed Systems
Fonseca, P., Zhang, K., Wang, X., & Krishnamurthy, A. (2017) · European Conference on Computer Systems · doi:10.1145/3064176.3064183 · Cited by 70 · see citing works
What Do We Know about Knowledge Management? Practical Implications for Software Engineering
Dingsøyr, T., Bjørnson, F. O., & Shull, F. J. (2009) · IEEE Software · doi:10.1109/ms.2009.82 · Cited by 59 · see citing works
SoK: Analysis of Software Supply Chain Security by Establishing Secure Design Properties
Okafor, C., Schorlemmer, T. R., Torres-Arias, S., & Davis, J. C. (2022) · doi:10.1145/3560835.3564556 · Cited by 56 · see citing works
Self-citationObject Oriented Software Modeling Using NLP Based Knowledge Extraction
Bajwa, I. S. (2009) · Cited by 56 · see citing works
Evaluating and Mitigating Software Supply Chain Security Risks
Ellison, R. J., Goodenough, J., Weinstock, C. B., & Woody, C. (2010) · KiltHub Repository · doi:10.1184/r1/6573497 · Cited by 42 · see citing works
Ernst, M. D. (2017) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.snapl.2017.4 · Cited by 40 · see citing works
A Systemic Approach for Assessing Software Supply-Chain Risk
Alberts, C., Dorofee, A., Creel, R., Ellison, R. J., & Woody, C. (2011) · Hawaii International Conference on System Sciences (HICSS) · doi:10.1109/hicss.2011.36 · Cited by 33 · see citing works
Supply-Chain Vulnerability Elimination via Active Learning and Regeneration
Vasilakis, N., Benetopoulos, A., Handa, S., Schoen, A., Shen, J., & Rinard, M. (2021) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3460120.3484736 · Cited by 24 · see citing works
Nissen, C. A., Gronager, J. E., Metzger, R. S., & Rishikof, H. (2018) · Cited by 24 · see citing works
Reusing Deep Learning Models: Challenges and Directions in Software Engineering
Davis, J. C., Jajal, P., Jiang, W., Schorlemmer, T. R., Synovic, N. M., & Thiruvathukal, G. K. (2023) · IEEE John Vincent Atanasoff International Symposium on Modern Computing (JVA) · doi:10.1109/jva60410.2023.00015 · Cited by 23 · see citing works
Self-citationHow VMware Exploits Contributed to SolarWinds Supply-chain Attack
Huddleston, J., Ji, P., Bhunia, S., & Cogan, J. (2021) · International Conference on Computational Science and Computational Intelligence (CSCI) · doi:10.1109/csci54926.2021.00190 · Cited by 23 · see citing works
Reflecting on Recurring Failures in IoT Development
Anandayuvaraj, D., & Davis, J. C. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3559545 · Cited by 20 · see citing works
Self-citationSoftware engineering under deadline pressure
Costello, S. H. (1984) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/1010941.1010947 · Cited by 19 · see citing works
Reflections on software failure analysis
Amusuo, P. C., Sharma, A., Rao, S. R., Vincent, A., & Davis, J. C. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3560879 · Cited by 16 · see citing works
Self-citationThe Pathology of Failures in IoT Systems
Melo, M., & de Aquino, G. S. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87013-3_33 · Cited by 7 · see citing works
Software supply chain: A taxonomy of attacks, mitigations and risk assessment strategies
Gokkaya, B., Aniello, L., & Halak, B. (2026) · Journal of Information Security and Applications · doi:10.1016/j.jisa.2025.104324 · Cited by 7 · see citing works
Anandayuvaraj, D., Thulluri, P., Figueroa, J., Shandilya, H., & Davis, J. C. (2023) · IEEE/ACM International Workshop on Software Engineering Research and Practices for the IoT (SERP4IoT) · doi:10.1109/serp4iot59158.2023.00011 · Cited by 6 · see citing works
Self-citationSoftware supply chain: review of attacks, risk assessment strategies and security controlsPreprint
Gokkaya, B., Aniello, L., & Halak, B. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.14157 · Cited by 4 · see citing works
Barriers for Post mortem evaluations in Systems Development
Pedersen, K. (2004) · Cited by 3 · see citing works
For Good Measure: Counting Broken Links: A Quant's View of Software Supply Chain Security.
Geer, D., Tozer, B., & Meyers, J. S. (2020) · Cited by 3 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
When LLMs meet cybersecurity: a systematic literature review
Zhang, J., Bu, H., Wen, H., Liu, Y., Fei, H., Xi, R., Li, L., Yang, Y., Zhu, H., & Meng, D. (2025) · Cybersecurity · doi:10.1186/s42400-025-00361-w · Cited by 220 · see citing works
“Jensen et al. (2024) compares the performance of a wide range of open-source and proprietary models with Python code snippets in assisting vulnerability discovery.”
Large language models in food science: Innovations, applications, and future
Ma, P., Tsai, S., He, Y., Jia, X., Zhen, D., Yu, N., Wang, Q., Ahuja, J. K. C., & Wei, C. (2024) · Trends in Food Science & Technology · doi:10.1016/j.tifs.2024.104488 · Cited by 97 · see citing works
Research Directions in Software Supply Chain Security
Williams, L. A., Benedetti, G., Hamer, S., Paramitha, R., Rahman, I., Tamanna, M., Tystahl, G., Zahan, N., Morrison, P. J., Acar, Y., et al. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3714464 · Cited by 74 · see citing works
Integrating large language models with internet of things: applications
Zong, M., Hekmati, A., Guastalla, M., Li, Y., & Krishnamachari, B. (2025) · Discover Internet of Things · doi:10.1007/s43926-024-00083-4 · Cited by 40 · see citing works
LLMs in Cyber Security: Bridging Practice and Education
Atlam, H. F. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9070184 · Cited by 27 · see citing works
Leveraging Large Language Models to Detect NPM Malicious Packages
Zahan, N., Burckhardt, P., Lysenko, M., Aboukhadijeh, F., & Williams, L. A. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00146 · Cited by 18 · see citing works
A systematic analysis of generative artificial intelligence for supply chain transformation
Bahroun, Z., Saihi, A., As’ad, R. A., & Tanash, M. (2025) · Supply Chain Analytics · doi:10.1016/j.sca.2025.100188 · Cited by 25 · see citing works
Generative AI and digital resilience: a research agenda
Große, C., & Sundberg, L. (2025) · Journal of Risk Research · doi:10.1080/13669877.2025.2539105 · Cited by 13 · see citing works
Exploring the Dual Role of LLMs in Cybersecurity: Threats and Defenses
Bryce, C., Kalousis, A., Leroux, I., Madinier, H., Pasche, T., & Ruch, P. (2024) · Springer eBooks · doi:10.1007/978-3-031-54827-7_26 · Cited by 6 · see citing works
Supply Chain Security, Technological Advancements, and Future Trends
Altamimi, S., & Al-Haija, Q. S. A. (2024) · Advances in information security, privacy, and ethics book series · doi:10.4018/979-8-3693-3451-5.ch010 · Cited by 5 · see citing works
HuggingGraph: Understanding the Supply Chain of LLM Ecosystem
Rahman, M. S., Gao, P., & Ji, Y. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761510 · Cited by 6 · see citing works
Exploring the Landscape of Programming Language Identification With Machine Learning Approaches
Verma, A., Saha, R., Kumar, G., Brighente, A., Conti, M., & Kim, T.-H. (2025) · IEEE Access · doi:10.1109/access.2025.3538108 · Cited by 4 · see citing works
CrowdStrike Causes Global Microsoft Outage
Alsowaigh, R. E. (2025) · Journal of Information Security and Cybercrimes Research · doi:10.26735/qhdd4798 · Cited by 4 · see citing works
FAIL: Analyzing Software Failures from the News Using LLMs
Anandayuvaraj, D., Campbell, M., Tewari, A., & Davis, J. C. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695022 · Cited by 6 · see citing works
Self-citationSecuring educational LLMs: A generalised taxonomy of attacks on LLMs and DREAD risk assessment
Zahid, F., Sewwandi, A., Brandon, L., Kumar, V., & Sinha, R. (2025) · High-Confidence Computing · doi:10.1016/j.hcc.2025.100371 · Cited by 4 · see citing works
Leveraging Large Language Models to Detect npm Malicious Packages
Zahan, N., Burckhardt, P., Lysenko, M., Aboukhadijeh, F., & Williams, L. (2024) · arXiv · doi:10.48550/arxiv.2403.12196 · Cited by 3 · see citing works
Basbas, M., Djamaa, B., & Senouci, M. R. (2026) · Security and Privacy · doi:10.1002/spy2.70223 · Cited by 1 · see citing works
Leveraging the Graph-Based LLM to Support the Analysis of Supply Chain Information
Su, P., Xu, R., & Chen, D. (2025) · Informatics · doi:10.3390/informatics12040124 · Cited by 1 · see citing works
Özbay, R., Çelebi, M., & Yavanoğlu, U. (2026) · IEEE Access · doi:10.1109/access.2026.3658308 · Cited by 3 · see citing works
The Impact of Large Language Models on Cybersecurity
Moongela, H., & Mayayise, T. O. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-96262-2_10 · Cited by 1 · see citing works
ZTD$_{JAVA}$: Mitigating Software Supply Chain Vulnerabilities via Zero-Trust DependenciesPreprint
Amusuo, P. C., Robinson, K. A., Singla, T., Peng, H., Machiry, A., Torres-Arias, S., Simon, L., & Davis, J. C. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.14117
Self-citationPlotting OSS-Based Supply Chain Attack Strategies and the Defense Failure
Sarang, A. D., Choi, S.-H., & Park, K.-W. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-1624-4_24
Cai, Z., Cai, Y., Leng, T., Zhou, H., Xie, X., Sun, Y., Chai, Y., & Qiu, J. (2025) · IEEE International Conference on Data Science in Cyberspace · doi:10.1109/dsc67331.2025.00044
Artificial Intelligence and Machine Learning for Supply Chain Security
Razzaq, K., & Abbasi, S. I. (2026) · Signals and communication technology · doi:10.1007/978-3-032-11119-7_7
Edge-to-Cloud DevSecOps Workflows: End-to-End Compliance Automation for Enterprise Data Engineering
Srivastava, S., Nellutla, N., Peddi, S., & Thaneeru, A. (2025) · IEEE Global Conference on Wireless Computing and Networking (GCWCN) · doi:10.1109/gcwcn66157.2025.11448336
Betancourt, A., COYAGO, G. A. A., Sangoluisa, K., Auz-Torres, M., & Yoo, S. G. (2026) · IEEE Access · doi:10.1109/access.2026.3718389
VeriCrypt-Agent: Evidence-Grounded Multi-Agent Verification for Cryptographic Dependency Updates
Тынченко, В. С., Бородулин, А. С., Martysyuk, D., Gantimurov, A., Nelyub, V., & Malashin, I. P. (2026) · Computers · doi:10.3390/computers15090638
- Authors
- Singla, T., Anandayuvaraj, D., Kalu, K. G., Schorlemmer, T. R., & Davis, J. C.
- Affiliations
- Purdue University West Lafayette (US)
- Venue
- Proceedings of the 2023 Workshop on Software Supply Chain Offensive Research and Ecosystem Defenses
- Publisher
- ACM
- Published
- 2023-11-26
- Pages
- 5-15
- OpenAlex
- W4388954904
- Cited by
- 26
- References
- 45
- Citations by year
- 2026: 5 · 2025: 16 · 2024: 4 · 2023: 1
- FWCI
- 6.68
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Workshop on Software Supply Chain Offensive Research and Ecosystem Defenses (SCORED) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- NSF (2229703)
- Cisco
- Topics
- Physical Sciences › Computer Science › Information Systems › Information and Cyber Security
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- software supply chain security, large language models, failure analysis, prompt engineering, automated security analysis, GPT-3.5, Google Bard, SolarWinds
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
