Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 77 of the 79 references it cites were found in a registry or an index.

Automated Infrastructure as Code Program Testing

Open access
Sokolowski, Daniel, Spielmann, David, Salvaneschi, Guido · 2024 · IEEE Transactions on Software Engineering

Abstract

Infrastructure as Code (IaC) enables efficient deployment and operation, which are crucial to releasing software quickly. As setups can be complex, developers implement IaC programs in general-purpose programming languages like TypeScript and Python, using PL-IaC solutions like Pulumi and AWS CDK. The reliability of such IaC programs is even more relevant than in traditional software because a bug in IaC impacts the whole system. Yet, even though testing is a standard development practice, it is rarely used for IaC programs. For instance, in August 2022, less than 1% of the public Pulumi IaC programs on GitHub implemented tests. Available IaC program testing techniques severely limit the development velocity or require much development effort.To solve these issues, we propose Automated Configuration Testing (ACT), a methodology to test IaC programs in many configurations quickly and with low effort. ACT automatically mocks all resource definitions in the IaC program and uses generator and oracle plugins for test generation and validation. We implement ACT inProTI, a testing tool for Pulumi TypeScript with a type-based generator and oracle, and support for application specifications. Our evaluation with 6 081 programs from GitHub and artificial benchmarks shows thatProTIcan directly be applied to existing IaC programs, quickly finds bugs where current techniques are infeasible, and enables reusing existing generators and oracles thanks to its pluggable architecture.

Overview

Integrity signals

Registry

Crossref · 10.1109/tse.2024.3393070

Cites

77 of 77 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

1 more claim about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Citations

Cites

Show all 79 references

Research works printed but not linked: 2

  1. Morris, 2021, Infrastructure as Code: Dynamic Systems for the Cloud Age
  2. Kim, 2021, The DevOps hHandbook: How to Create World-class Agility, Rel., & Secur. Technol. Organizations

Cited by

See all 20

Metadata

Authors
Sokolowski, D., Spielmann, D., & Salvaneschi, G.
Affiliations
University of St.Gallen (CH)
Venue
IEEE Transactions on Software Engineering
Published
2024-06
Volume
50
Issue
6
Pages
1585-1599
ISSN
  • 0098-5589 (ISSN-L)
  • 1939-3520
  • 2326-3881
OpenAlex
W4396564044
Cited by
20
References
79
Citations by year
2026: 12 · 2025: 5 · 2024: 3
FWCI
20.51
Type
journal article
Language
en
Open access
Locations
  • IEEE Transactions on Software Engineering (publishedVersion)
  • Alexandria (UniSG) (University of St.Gallen) (submittedVersion, other-oa)
Funders
Swiss National Science Foundation (200429)
Topics
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
Keywords
infrastructure as code, Pulumi, TypeScript, test case generation, cloud infrastructure reliability, bug detection

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Automated Infrastructure as Code Program Testing. https://app.cochat.ai/papers/10.1109/tse.2024.3393070
The paper
Sokolowski, D., Spielmann, D., & Salvaneschi, G. (2024). Automated Infrastructure as Code Program Testing. IEEE Transactions on Software Engineering. https://doi.org/10.1109/tse.2024.3393070

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.