The DOI matches its Crossref record; 77 of the 79 references it cites were found in a registry or an index.
Automated Infrastructure as Code Program Testing
Open accessAbstract
Infrastructure as Code (IaC) enables efficient deployment and operation, which are crucial to releasing software quickly. As setups can be complex, developers implement IaC programs in general-purpose programming languages like TypeScript and Python, using PL-IaC solutions like Pulumi and AWS CDK. The reliability of such IaC programs is even more relevant than in traditional software because a bug in IaC impacts the whole system. Yet, even though testing is a standard development practice, it is rarely used for IaC programs. For instance, in August 2022, less than 1% of the public Pulumi IaC programs on GitHub implemented tests. Available IaC program testing techniques severely limit the development velocity or require much development effort.To solve these issues, we propose Automated Configuration Testing (ACT), a methodology to test IaC programs in many configurations quickly and with low effort. ACT automatically mocks all resource definitions in the IaC program and uses generator and oracle plugins for test generation and validation. We implement ACT inProTI, a testing tool for Pulumi TypeScript with a type-based generator and oracle, and support for application specifications. Our evaluation with 6 081 programs from GitHub and artificial benchmarks shows thatProTIcan directly be applied to existing IaC programs, quickly finds bugs where current techniques are infeasible, and enables reusing existing generators and oracles thanks to its pluggable architecture.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In operations, knowing when to stop is part of the safety case [84], [85], [71].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [85]
In the citing paper
“An agent may identify the right cause and still fail operationally if it cannot show evidence, pass checks, prepare rollback, or leave an audit trail [85], [43], [74].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [85]
In the citing paper
“In guarded planner-executor systems, the human does not review a chat transcript.They review a bundle that can be scored, archived, and replayed [85], [70], [102], [74].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [85]
In the citing paper
“Autonomy changes which failures dominate [85], [43], [31].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [85]
In the citing paper
“trust in automation remains contextual and can be weakened by inconsistent or opaque behaviour [85].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [85]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Alloy: a lightweight object modelling notation
Jackson, D. (2002) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/505145.505149 · Cited by 1,181
The Daikon system for dynamic detection of likely invariants
Ernst, M. D., Perkins, J. H., Guo, P. J., McCamant, S., Pacheco, C., Tschantz, M. S., & Xiao, C. (2007) · Science of Computer Programming · doi:10.1016/j.scico.2007.01.015 · Cited by 1,080
Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation
Humble, J., & Farley, D. R. (2010) · Cited by 911
Feedback-Directed Random Test Generation
Pacheco, C., Lahiri, S. K., Ernst, M. D., & Ball, T. (2007) · 29th International Conference on Software Engineering (ICSE'07) · doi:10.1109/icse.2007.37 · Cited by 863
Software fault interactions and implications for software testing
Kuhn, D. R., Wallace, D. R., & Gallo, A. M. (2004) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2004.24 · Cited by 789
(2000) · doi:10.1145/351240.351266 · Cited by 753
A Survey of DevOps Concepts and Challenges
Leite, L., Rocha, C., Kon, F., Milojicic, D., & Meirelles, P. (2020) · ACM Computing Surveys · doi:10.1145/3359981 · Cited by 484
Li, J., Zhao, B., & Zhang, C. (2018) · Cybersecurity · doi:10.1186/s42400-018-0002-y · Cited by 278
Olsson, H. H., Alahyari, H., & Bosch, J. (2012) · 2012 38th Euromicro Conference on Software Engineering and Advanced Applications · doi:10.1109/seaa.2012.54 · Cited by 243
The Seven Sins: Security Smells in Infrastructure as Code Scripts
Rahman, A., Parnin, C., & Williams, L. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00033 · Cited by 228
Research works printed but not linked: 2
- Morris, 2021, Infrastructure as Code: Dynamic Systems for the Cloud Age
- Kim, 2021, The DevOps hHandbook: How to Create World-class Agility, Rel., & Secur. Technol. Organizations
Cited by
Drosos, G.-P., Sotiropoulos, T., Alexopoulos, G., Mitropoulos, D., & Su, Z. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3689799 · Cited by 31
Vulnerabilities in infrastructure as code: what, how many, and who?
War, A., Diallo, A., Habib, A., Klein, J., & Bissyandé, T. F. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10672-8 · Cited by 24
Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety
Bilal, M., Crowcroft, J., Wang, R., Xu, X., & Dustdar, S. (2027) · Computer Science Review · doi:10.1016/j.cosrev.2026.101075 · Cited by 10
“A diagnosis often implies a change, but a change must be expressed as a structured proposal, checked outside the model, and applied with limited permissions [87], [88], [70], [68], [69].”
InfraFix: Technology-Agnostic Repair of Infrastructure as Code
Saavedra, N., Ferreira, J. F., & Mendes, A. S. (2025) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3713081.3731735 · Cited by 6
An empirical study of policy as code: adoption, purpose, and maintenance
Opdebeeck, R., Alfadel, M., Rahman, A., Kashiwa, Y., Ferreira, J. F., Kula, R. G., & De Roover, C. (2026) · Proceedings of the 23rd International Conference on Mining Software Repositories · doi:10.1145/3793302.3793355 · Cited by 5
Metadata
- Authors
- Sokolowski, D., Spielmann, D., & Salvaneschi, G.
- Affiliations
- University of St.Gallen (CH)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2024-06
- Volume
- 50
- Issue
- 6
- Pages
- 1585-1599
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- 2326-3881
- OpenAlex
- W4396564044
- Cited by
- 20
- References
- 79
- Citations by year
- 2026: 12 · 2025: 5 · 2024: 3
- FWCI
- 20.51
- Type
- journal article
- Language
- en
- Open access
- Licence
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- Alexandria (UniSG) (University of St.Gallen) (submittedVersion, other-oa)
- Funders
- Swiss National Science Foundation (200429)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- infrastructure as code, Pulumi, TypeScript, test case generation, cloud infrastructure reliability, bug detection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Automated Infrastructure as Code Program Testing. https://app.cochat.ai/papers/10.1109/tse.2024.3393070
- The paper
- Sokolowski, D., Spielmann, D., & Salvaneschi, G. (2024). Automated Infrastructure as Code Program Testing. IEEE Transactions on Software Engineering. https://doi.org/10.1109/tse.2024.3393070
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.