Automated Infrastructure as Code Program Testing
Open accessThe DOI matches its Crossref record; 77 of the 79 references it cites were found in a registry or an index.
Abstract
Infrastructure as Code (IaC) enables efficient deployment and operation, which are crucial to releasing software quickly. As setups can be complex, developers implement IaC programs in general-purpose programming languages like TypeScript and Python, using PL-IaC solutions like Pulumi and AWS CDK. The reliability of such IaC programs is even more relevant than in traditional software because a bug in IaC impacts the whole system. Yet, even though testing is a standard development practice, it is rarely used for IaC programs. For instance, in August 2022, less than 1% of the public Pulumi IaC programs on GitHub implemented tests. Available IaC program testing techniques severely limit the development velocity or require much development effort.To solve these issues, we propose Automated Configuration Testing (ACT), a methodology to test IaC programs in many configurations quickly and with low effort. ACT automatically mocks all resource definitions in the IaC program and uses generator and oracle plugins for test generation and validation. We implement ACT inProTI, a testing tool for Pulumi TypeScript with a type-based generator and oracle, and support for application specifications. Our evaluation with 6 081 programs from GitHub and artificial benchmarks shows thatProTIcan directly be applied to existing IaC programs, quickly finds bugs where current techniques are infeasible, and enables reusing existing generators and oracles thanks to its pluggable architecture.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In operations, knowing when to stop is part of the safety case [84], [85], [71].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [85]
In the citing paper
“An agent may identify the right cause and still fail operationally if it cannot show evidence, pass checks, prepare rollback, or leave an audit trail [85], [43], [74].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [85]
In the citing paper
“In guarded planner-executor systems, the human does not review a chat transcript.They review a bundle that can be scored, archived, and replayed [85], [70], [102], [74].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [85]
In the citing paper
“Autonomy changes which failures dominate [85], [43], [31].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [85]
In the citing paper
“trust in automation remains contextual and can be weakened by inconsistent or opaque behaviour [85].”
From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [85]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Alloy: a lightweight object modelling notation
Jackson, D. (2002) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/505145.505149 · Cited by 1,181 · cited 1×
“E.g., Alloy [83] is a language and analysis tool to verify structural properties of software.”
MethodThe Daikon system for dynamic detection of likely invariants
Ernst, M. D., Perkins, J. H., Guo, P. J., McCamant, S., Pacheco, C., Tschantz, M. S., & Xiao, C. (2007) · Science of Computer Programming · doi:10.1016/j.scico.2007.01.015 · Cited by 1,080 · cited 2×
“Daikon[44] is a dynamic invariant detector that identifies application invariants in a set of program traces.”
BackgroundMethodContinuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation
Humble, J., & Farley, D. R. (2010) · Cited by 911
Feedback-Directed Random Test Generation
Pacheco, C., Lahiri, S. K., Ernst, M. D., & Ball, T. (2007) · 29th International Conference on Software Engineering (ICSE'07) · doi:10.1109/icse.2007.37 · Cited by 863 · cited 1×
“As the generator plugin is exchangeable, ACT is amenable to new state-of-the-art fuzzing and PBT test case selection strategies, e.g., based on testing feedback [31], search-based techniques [32], code coverage [33], and combinatorial coverage [34].”
MethodSoftware fault interactions and implications for software testing
Kuhn, D. R., Wallace, D. R., & Gallo, A. M. (2004) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2004.24 · Cited by 789 · cited 1×
“Kuhn et al. [100] found that most bugs are caused by the interaction of only a few parameters, motivating combinatorial testing [101], which Goldstein et al. [34] applied to PBT generators by modifying the random generator distributions.”
Background(2000) · doi:10.1145/351240.351266 · Cited by 753 · cited 4×
“We determined whether ProTI can find bugs quickly and reliably compared to existing PL-IaC testing techniques (cf. Section II-B).”
BackgroundMethodInfluentialA Survey of DevOps Concepts and Challenges
Leite, L., Rocha, C., Kon, F., Milojicic, D., & Meirelles, P. (2020) · ACM Computing Surveys · doi:10.1145/3359981 · Cited by 484 · cited 1×
“As a result, IaC enables faster, more reproducible software operations [3], [4], [5], [6].”
MethodLi, J., Zhao, B., & Zhang, C. (2018) · Cybersecurity · doi:10.1186/s42400-018-0002-y · Cited by 278
Olsson, H. H., Alahyari, H., & Bosch, J. (2012) · 2012 38th Euromicro Conference on Software Engineering and Advanced Applications · doi:10.1109/seaa.2012.54 · Cited by 243 · cited 2×
“Mocking code easily becomes more complex than the IaC program under test itself, resulting in very few projects using systematic testing—despite testing being crucial for the high-velocity development of reliable software [17], [18].”
BackgroundThe Seven Sins: Security Smells in Infrastructure as Code Scripts
Rahman, A., Parnin, C., & Williams, L. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00033 · Cited by 228 · cited 1×
“Restrictions apply. smells and proposed linters for Puppet, Ansible, and Chef [56], [57].”
BackgroundPractical Combinatorial Testing: Beyond Pairwise
Kuhn, R., Lei, Y., & Kacker, R. (2008) · IT Professional · doi:10.1109/mitp.2008.54 · Cited by 194 · cited 1×
“Kuhn et al. [100] found that most bugs are caused by the interaction of only a few parameters, motivating combinatorial testing [101], which Goldstein et al. [34] applied to PBT generators by modifying the random generator distributions.”
BackgroundA systematic mapping study of infrastructure as code research
Rahman, A., Mahdavi-Hezaveh, R., & Williams, L. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2018.12.004 · Cited by 174
Does your configuration code smell?
Sharma, T., Fragkoulis, M., & Spinellis, D. (2016) · Proceedings of the 13th International Conference on Mining Software Repositories · doi:10.1145/2901739.2901761 · Cited by 157 · cited 1×
“Sharma et al. [53] were the first to identify code smells in Puppet scripts.”
BackgroundProperty-based testing: a new approach to testing for assurance
Fink, G., & Bishop, M. (1997) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/263244.263267 · Cited by 144 · cited 2×
“Conceptually, ACT combines property-based testing (PBT) [28], [29] and fuzzing [30] techniques for IaC programs.”
MethodArchJava: connecting software architecture to implementation
Aldrich, J., Chambers, C., & Notkin, D. (2003) · Proceedings of the 24th International Conference on Software Engineering. ICSE 2002 · doi:10.1109/icse.2002.1007967 · Cited by 141 · cited 1×
“For example, ArchJava [73] embeds such specifications in Java source code, enabling architecture compliance checks at compile time.”
BackgroundAdoption, Support, and Challenges of Infrastructure-as-Code: Insights from Industry
Guerriero, M., Garriga, M., Tamburri, D. A., & Palomba, F. (2019) · 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME) · doi:10.1109/icsme.2019.00092 · Cited by 140 · cited 3×
“Kumara et al. [4] and Guerriero et al. [3] explored IaC best practices and issues in the industry through a grey literature survey and practitioner interviews.”
BackgroundMethodSemantic-based Automated Reasoning for AWS Access Policies using SMT
Backes, J., Bolignano, P., Cook, B., Dodge, C., Gacek, A., Luckow, K., Rungta, N., Tkachuk, O., & Varming, C. (2018) · 2018 Formal Methods in Computer Aided Design (FMCAD) · doi:10.23919/fmcad.2018.8602994 · Cited by 106 · cited 1×
“AWS investigated automatically verifying infrastructure properties [78], [79], [80], leading to at least two automated services in production: AWS Tiros verifies reachability queries on virtual networks [81] and AWS Zelkova performs access ver-ification on role-based AWS IAM policies [82].”
MethodFormal Reasoning About the Security of Amazon Web Services
Cook, B. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-96145-3_3 · Cited by 105 · cited 1×
“AWS investigated automatically verifying infrastructure properties [78], [79], [80], leading to at least two automated services in production: AWS Tiros verifies reachability queries on virtual networks [81] and AWS Zelkova performs access ver-ification on role-based AWS IAM policies [82].”
MethodAssessing the Construct Validity of Object-Oriented, Class-Level Code Quality Metrics
Ralph, P., bin Ali, N., Baltes, S., Bianculli, D., Díaz, J., Dittrich, Y., Ernst, N., Felderer, M., FELDT, R. H., Filieri, A., et al. (2020) · arXiv (Cornell University) · doi:10.4230/lipics.esem.2026.10 · Cited by 97
Testing Idempotence for Infrastructure as Code
Hummer, W., Rosenberg, F., Oliveira, F., & Eilam, T. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-45065-5_19 · Cited by 92
To Mock or Not to Mock? An Empirical Study on Mocking Practices
Spadini, D., Aniche, M. F., Bruntink, M., & Bacchelli, A. (2017) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2017.61 · Cited by 92
An Empirical Study of Adoption of Software Testing in Open Source Projects
Kochhar, P. S., Bissyandé, T. F., Lo, D., & Jiang, L. (2013) · Asia-Pacific Conference on Quality Software (APAQS) · doi:10.1109/qsic.2013.57 · Cited by 91
The essential deployment metamodel: a systematic review of deployment automation technologies
Wurster, M., Breitenbücher, U., Falkenthal, M., Krieger, C., Leymann, F., Saatkamp, K., & Soldani, J. (2019) · SICS Software-Intensive Cyber-Physical Systems · doi:10.1007/s00450-019-00412-x · Cited by 88 · cited 1×
“Wurster et al. [71] presented a systematic review of declarative deployment technologies and introduced a metamodel for their common core.”
BackgroundCombining Declarative and Imperative Cloud Application Provisioning Based on TOSCA
Breitenbücher, U., Binz, T., Képes, K., Kopp, O., Leymann, F., & Wettinger, J. (2014) · IEEE International Conference on Cloud Engineering · doi:10.1109/ic2e.2014.56 · Cited by 86 · cited 1×
“With The authors are with the University of St. Gallen, 9000 St. Gallen, Switzer-land (e-mail: daniel.sokolowski@unisg.ch; david.spielmann@unisg.ch; guido.salvaneschi@unisg.ch).”
BackgroundTaneja, K., Zhang, Y., & Xie, T. (2010) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/1858996.1859053 · Cited by 83 · cited 1×
“Taneja et al. [89] proposed MODA, using an efficient, SQL-aware mock and advanced test generation techniques to automatically test database applications.”
MethodWithin-Project Defect Prediction of Infrastructure-as-Code Using Product and Process Metrics
Dalla Palma, S., Di Nucci, D., Palomba, F., & Tamburri, D. A. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3051492 · Cited by 83 · cited 1×
“Dalla Palma et al. [63], [64], [65] proposed various quality metrics and an AI defect prediction framework for Ansible scripts.”
BackgroundSecurity Smells in Ansible and Chef Scripts: A Replication Study
Rahman, A., Rahman, M. R., Parnin, C., & Williams, L. (2021) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3408897 · Cited by 80 · cited 1×
“Restrictions apply. smells and proposed linters for Puppet, Ansible, and Chef [56], [57].”
BackgroundToward a catalog of software quality metrics for infrastructure code
Dalla Palma, S., Di Nucci, D., Palomba, F., & Tamburri, D. A. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110726 · Cited by 79
Steinhöfel, D., & Zeller, A. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549139 · Cited by 75 · cited 1×
“Yet, input-value-generation-guided approaches exist; for example, grammar-based fuzzing is an active research field [95], [96].”
BackgroundFoundational Property-Based Testing
Paraskevopoulou, Z., Hriţcu, C., Dénès, M., Lampropoulos, L., & Pierce, B. C. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-22102-1_22 · Cited by 72 · cited 1×
“On the intersection with formal methods, Paraskevopoulou et al. [102] integrated PBT into a proof assistant to verify tests, and Lampropoulos et al. [103] compiled logical conditions (inductive relations) to generators and to their soundness and completeness proofs.”
MethodSource code properties of defective infrastructure as code scripts
Rahman, A., & Williams, L. A. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2019.04.013 · Cited by 72
The do’s and don’ts of infrastructure code: A systematic gray literature review
Kumara, I., Garriga, M., Romeu, A. U., Di Nucci, D., Palomba, F., Tamburri, D. A., & van den Heuvel, W.-J. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106593 · Cited by 70 · cited 2×
“Kumara et al. [4] and Guerriero et al. [3] explored IaC best practices and issues in the industry through a grey literature survey and practitioner interviews.”
BackgroundMethodMethods and Tools for Policy Analysis
Jabal, A. A., Davari, M., Bertino, E., Makaya, C., Calo, S. B., Verma, D. C., Russo, A. M., & Williams, C. B. (2019) · ACM Computing Surveys · doi:10.1145/3295749 · Cited by 68 · cited 1×
“Lastly, Abu Jabal et al. [86] gave a comprehensive overview of techniques for policy verification, focused on access control and network management.”
BackgroundGLITCH: Automated Polyglot Security Smell Detection in Infrastructure as Code
Saavedra, N., & Ferreira, J. F. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556945 · Cited by 68 · cited 1×
“Saavedra and Ferreira [58] introduced GLITCH for linters on a CaC-solution-agnostic intermediate representation.”
BackgroundRehearsal: a configuration verification tool for puppet
Shambaugh, R., Weiss, A., & Guha, A. (2016) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/2908080.2908083 · Cited by 66 · cited 1×
“Shambaugh et al. [52] proposed Rehearsal to verify the determinacy and idempotency of Puppet scripts.”
BackgroundReachability Analysis for AWS-Based Networks
Backes, J., Bayless, S., Cook, B., Dodge, C., Gacek, A., Hu, A. J., Kahsai, T., Kocik, B., Kotelnikov, E., Kukovec, J., et al. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-25543-5_14 · Cited by 66 · cited 1×
“AWS investigated automatically verifying infrastructure properties [78], [79], [80], leading to at least two automated services in production: AWS Tiros verifies reachability queries on virtual networks [81] and AWS Zelkova performs access ver-ification on role-based AWS IAM policies [82].”
MethodGang of eight: a defect taxonomy for infrastructure as code scripts
Rahman, A., Farhana, E., Parnin, C., & Williams, L. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380409 · Cited by 65 · cited 1×
“In contrast to previous, more fine-grained bug taxonomies, e.g., for IaC defects by Rahman et al. [35], we focus purely on the required oracle to find a bug.”
ResultBeginner's luck: a language for property-based generators
Lampropoulos, L., Gallois-Wong, D., Hriţcu, C., Hughes, J., Pierce, B. C., & Xia, L.-Y. (2017) · ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL) · doi:10.1145/3009837.3009868 · Cited by 64 · cited 1×
“Lampropoulos et al. proposed Luck, a language for PBT generators [97], and coverage-guided PBT [98].”
BackgroundA dependency constraint language to manage object‐oriented software architectures
Terra, R., & Valente, M. T. (2009) · Software: Practice and Experience · doi:10.1002/spe.931 · Cited by 61 · cited 1×
“Terra and de Oliveira Valente [75] proposed specifying and statically enforcing dependencies in the software architectures to avoid erosion.”
BackgroundFully automated functional fuzzing of Android apps for detecting non-crashing logic bugs
Su, T., Yan, Y., Wang, J., Sun, J., Xiong, Y., Pu, G., Wang, K., & Su, Z. (2021) · Proceedings of the ACM on Programming Languages · doi:10.1145/3485533 · Cited by 60 · cited 1×
“Recent fuzzing literature, e.g., Su et al. [36] and Li et al. [33], commonly distinguishes crash bugs that cause the program to crash and non-crashing logic bugs , which require a more precise oracle than crash detection to identify erroneous computations.”
BackgroundCode Smells in Infrastructure as Code
Schwarz, J., Steffens, A., & Lichter, H. (2018) · 2018 11th International Conference on the Quality of Information and Communications Technology (QUATIC) · doi:10.1109/quatic.2018.00040 · Cited by 59 · cited 1×
“Later studies confirmed them for Chef [54].”
Performance modelling and verification of cloud-based auto-scaling policies
Evangelidis, A., Parker, D., & Rami Bahsoon, A. (2018) · Future Generation Computer Systems · doi:10.1016/j.future.2017.12.047 · Cited by 56 · cited 1×
“Evangelidis et al. [85] proposed probabilistic verification of performance properties of rule-based auto-scaling policies.”
BackgroundTargeted property-based testing
Löscher, A., & Sagonas, K. (2017) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3092703.3092711 · Cited by 54 · cited 2×
“This way, each resource input configuration C I is run through validations and transformations that the provider’s SDK may implement in the resources’ constructors.”
BackgroundMethodSCARPE: A Technique and Tool for Selective Capture and Replay of Program Executions
Joshi, S., & Orso, A. (2007) · Proceedings/Proceedings - Conference on Software Maintenance · doi:10.1109/icsm.2007.4362636 · Cited by 51 · cited 1×
“Various works synthesize mocks from interaction traces of components [91], [92], [93].”
BackgroundCoverage guided, property based testing
Lampropoulos, L., Hicks, M., & Pierce, B. C. (2019) · Proceedings of the ACM on Programming Languages · doi:10.1145/3360607 · Cited by 49 · cited 1×
“Lampropoulos et al. proposed Luck, a language for PBT generators [97], and coverage-guided PBT [98].”
BackgroundGenerating good generators for inductive relations
Lampropoulos, L., Paraskevopoulou, Z., & Pierce, B. C. (2018) · Proceedings of the ACM on Programming Languages · doi:10.1145/3158133 · Cited by 44 · cited 1×
“On the intersection with formal methods, Paraskevopoulou et al. [102] integrated PBT into a proof assistant to verify tests, and Lampropoulos et al. [103] compiled logical conditions (inductive relations) to generators and to their soundness and completeness proofs.”
MethodSmelly variables in ansible infrastructure code: detection, prevalence, and lifetime
Opdebeeck, R., Zerouali, A., & Roover, C. D. (2022) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3524842.3527964 · Cited by 43 · cited 1×
“Opdebeeck et al. [60], [61] analyzed the quality of semantic versioning and variable-precedence-related code smells in An-sible.”
BackgroundOpdebeeck, R., Zerouali, A., & Roover, C. D. (2023) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr59073.2023.00079 · Cited by 43 · cited 1×
“Further, they applied program dependence graph analysis to Ansible scripts, motivating control-and data-flow analysis for IaC security smell detection techniques [62].”
MethodDistributed system development with ScalaLoci
Weisenburger, P., Köhler, M., & Salvaneschi, G. (2018) · Proceedings of the ACM on Programming Languages · doi:10.1145/3276499 · Cited by 39 · cited 1×
“Placement types are a language approach where the type system checks architectural conformance [76], [77].”
BackgroundSelf-citationAutomating serverless deployments for DevOps organizations
Sokolowski, D., Weisenburger, P., & Salvaneschi, G. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468575 · Cited by 36 · cited 1×
“Research on PL-IaC has been limited to deployment coordination [48], [49].”
BackgroundSelf-citationBlock public access: trust safety verification of access control policies
Bouchet, M., Cook, B., Cutler, B., Druzkina, A., Gacek, A., Hadarean, L., Jhala, R., Marshall, B., Peebles, D., Rungta, N., et al. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3409728 · Cited by 35 · cited 1×
“AWS investigated automatically verifying infrastructure properties [78], [79], [80], leading to at least two automated services in production: AWS Tiros verifies reachability queries on virtual networks [81] and AWS Zelkova performs access ver-ification on role-based AWS IAM policies [82].”
MethodMock object creation for test factoring
Saff, D., & Ernst, M. D. (2004) · ACM SIGPLAN-SIGSOFT Workshop on Program Analysis for Software Tools and Engineering (PASTE) · doi:10.1145/996821.996838 · Cited by 34 · cited 1×
“Various works synthesize mocks from interaction traces of components [91], [92], [93].”
BackgroundSpecification of cloud topologies and orchestration using TOSCA: a survey
Bellendorf, J., & Mann, Z. Á. (2019) · Computing · doi:10.1007/s00607-019-00750-3 · Cited by 32 · cited 1×
“Bellendorf and Mann [70] surveyed existing literature on TOSCA.”
BackgroundStatic Analysis of Infrastructure as Code: a Survey
Chiari, M., Pascalis, M. D., & Pradella, M. (2022) · IEEE International Conference on Software Architecture Companion (ICSA-C) · doi:10.1109/icsa-c54293.2022.00049 · Cited by 32 · cited 1×
“Borovits et al. [67] proposed FindICI, an AI-based tool to identify linguistic inconsistency between documentation, comments, and code in Ansible scripts, and Chiari et al. [68] surveyed work on static analysis for IaC, focusing mainly on CaC.”
MethodBackes, J., Bolignano, P., Cook, B., Gacek, A., Luckow, K. S., Rungta, N., Schäf, M., Schlesinger, C., Tanash, R. S., Varming, C., et al. (2019) · IEEE Software · doi:10.1109/ms.2019.2930609 · Cited by 31 · cited 1×
“AWS investigated automatically verifying infrastructure properties [78], [79], [80], leading to at least two automated services in production: AWS Tiros verifies reachability queries on virtual networks [81] and AWS Zelkova performs access ver-ification on role-based AWS IAM policies [82].”
MethodBorovits, N., Kumara, I., Nucci, D. D., Krishnan, P., Palma, S. D., Palomba, F., Tamburri, D. A., & van den Heuvel, W. (2022) · Empirical Software Engineering · doi:10.1007/s10664-022-10215-5 · Cited by 30 · cited 1×
“Borovits et al. [67] proposed FindICI, an AI-based tool to identify linguistic inconsistency between documentation, comments, and code in Ansible scripts, and Chiari et al. [68] surveyed work on static analysis for IaC, focusing mainly on CaC.”
MethodAnsibleMetrics: A Python library for measuring Infrastructure-as-Code blueprints in Ansible
Palma, S. D., Nucci, D. D., & Tamburri, D. A. (2020) · SoftwareX · doi:10.1016/j.softx.2020.100633 · Cited by 29 · cited 1×
“Further, they applied program dependence graph analysis to Ansible scripts, motivating control-and data-flow analysis for IaC security smell detection techniques [62].”
MethodAs Code Testing: Characterizing Test Quality in Open Source Ansible Development
Hassan, M. M., & Rahman, A. (2022) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst53961.2022.00031 · Cited by 29 · cited 1×
“Hassan and Rahman [66] studied bugs in open-source Ansible test scripts.”
BackgroundLeveraging Practitioners’ Feedback to Improve a Security Linter
Reis, S., Abreu, R., d’Amorim, M., & Fortunato, D. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3560419 · Cited by 28 · cited 1×
“Reis et al. [59] found that such linters are too imprecise but can be improved through user feedback.”
BackgroundAutomating Targeted Property-Based Testing
Löscher, A., & Sagonas, K. (2018) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2018.00017 · Cited by 26 · cited 1×
“Lampropoulos et al. proposed Luck, a language for PBT generators [97], and coverage-guided PBT [98].”
BackgroundDo Judge a Test by its Cover: Combining Combinatorial and Property-Based Testing
Goldstein, H., Hughes, J., Lampropoulos, L., & Pierce, B. C. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-72019-3_10 · Cited by 24 · cited 2×
“Kuhn et al. [100] found that most bugs are caused by the interaction of only a few parameters, motivating combinatorial testing [101], which Goldstein et al. [34] applied to PBT generators by modifying the random generator distributions.”
BackgroundMethodStubCoder : Automated Generation and Repair of Stub Code for Mock Objects
Zhu, H., Wei, L., Terragni, V., Liu, Y., Cheung, S.-C., Wu, J., Sheng, Q., Zhang, B., & Song, L. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3617171 · Cited by 23 · cited 1×
“In contrast, Zhu et al.’s StubCoder [94] synthesizes mocks for regression testing solely from the tests’ code, without running the mocked component.”
MethodWurster, M., Breitenbücher, U., Harzenetter, L., Leymann, F., Soldani, J., & Yussupov, V. (2020) · International Conference on Cloud Computing and Services Science · doi:10.5220/0009794302160226 · Cited by 22 · cited 1×
“A follow-up work [72] leverages such core to define TOSCA Light, a subset of TOSCA, aiming to reconcile research modeling and industrial practice.”
BackgroundType-Checking CRDT Convergence
Zakhour, G., Weisenburger, P., & Salvaneschi, G. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3591276 · Cited by 19
Self-citationGrowing a Test Corpus with Bonsai Fuzzing
Vikram, V., Padhye, R., & Sen, K. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00072 · Cited by 17 · cited 1×
“Yet, input-value-generation-guided approaches exist; for example, grammar-based fuzzing is an active research field [95], [96].”
BackgroundOpdebeeck, R., Zerouali, A., Velázquez-Rodríguez, C., & Roover, C. D. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111059 · Cited by 17 · cited 1×
“Reis et al. [59] found that such linters are too imprecise but can be improved through user feedback.”
BackgroundA framework for automated test mocking of mobile apps
Fazzini, M., Gorla, A., & Orso, A. (2020) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3324884.3418927 · Cited by 16 · cited 1×
“Various works synthesize mocks from interaction traces of components [91], [92], [93].”
BackgroundTest Suite Reduction in Idempotence Testing of Infrastructure as Code
Ikeshita, K., Ishikawa, F., & Honiden, S. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-61467-0_6 · Cited by 15 · cited 1×
“Hummer et al. [50] proposed an idempotency testing approach for Chef scripts, which Ikeshita et al. [51] augmented with verification techniques to minimize the size of the required test suite.”
MethodThe PIPr Dataset of Public Infrastructure as Code Programs
Sokolowski, D., Spielmann, D., & Salvaneschi, G. (2024) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3643991.3644888 · Cited by 9 · cited 8×
“…lack of suitable testing techniques is especially apparent for PL-IaC: while studies found that more than 50 % of public software projects on GitHub use testing [14], [15], we found only 25 % of the PL-IaC programs use testing, dropping to 1 % for general PL-IaC [16], which only Pulumi implements.”
BackgroundMethodInfluentialSelf-citationProperty-Based Test Case Generators for Free
Angelis, E. D., Fioravanti, F., Palacios, A., Pettorossi, A., & Proietti, M. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-31157-5_12 · Cited by 8 · cited 1×
“On the intersection with formal methods, Paraskevopoulou et al. [102] integrated PBT into a proof assistant to verify tests, and Lampropoulos et al. [103] compiled logical conditions (inductive relations) to generators and to their soundness and completeness proofs.”
MethodEmpirical Study of Test Case and Test Framework Presence in Public Projects on GitHub
Madeja, M., Porubän, J., Chodarev, S., Sulír, M., & Gurbáľ, F. (2021) · Applied Sciences · doi:10.3390/app11167250 · Cited by 8 · cited 1×
“…lack of suitable testing techniques is especially apparent for PL-IaC: while studies found that more than 50 % of public software projects on GitHub use testing [14], [15], we found only 25 % of the PL-IaC programs use testing, dropping to 1 % for general PL-IaC [16], which only Pulumi implements.”
MethodContract-Based Mocking for Services-Oriented Development
Solms, F., & Marshall, L. (2016) · Annual Conference of the South African Institute of Computer Scientists and Information Technologists (SAICSIT) · doi:10.1145/2987491.2987534 · Cited by 6 · cited 1×
“Solms and Marshall [90] automatically generated mocks from explicit component contracts.”
BackgroundReasoning about distributed reconfigurable systems
Ahrens, E., Bozga, M. D., Iosif, R., & Katoen, J.-P. (2022) · Proceedings of the ACM on Programming Languages · doi:10.1145/3563293 · Cited by 6 · cited 1×
“Ahrens et al. [84] developed a proof system for invariants on reconfigurable distributed systems.”
BackgroundDecentralizing Infrastructure as Code
Sokolowski, D., Weisenburger, P., & Salvaneschi, G. (2022) · IEEE Software · doi:10.1109/ms.2022.3192968 · Cited by 5
Self-citationDynamic Service Composition and Deployment with OpenRichServices
Krüger, I. H., Demchak, B., & Menarini, M. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-30835-2_9 · Cited by 4 · cited 1×
“Krüger et al. [74] introduced ORS for the compositional specification, deployment, and dynamic reconfiguration of systems of services.”
BackgroundType-Safe Dynamic Placement with First-Class Placed Values
Zakhour, G., Weisenburger, P., & Salvaneschi, G. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3622873 · Cited by 2 · cited 1×
“ProTI verifies the correct composition of infrastructure configuration, e.g., through type-based oracles, and enables application-specific checks through ad-hoc specifications.”
Self-citationProTI: Automated Unit Testing of Pulumi TypeScript Infrastructure as Code Programs
Sokolowski, D., Spielmann, D., & Salvaneschi, G. (2023) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.10028479 · Cited by 2
Self-citation
Research works printed but not linked: 2
- Morris, 2021, Infrastructure as Code: Dynamic Systems for the Cloud Age
- Kim, 2021, The DevOps hHandbook: How to Create World-class Agility, Rel., & Secur. Technol. Organizations
Cited by
Drosos, G.-P., Sotiropoulos, T., Alexopoulos, G., Mitropoulos, D., & Su, Z. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3689799 · Cited by 31
“…between Puppet resources, and cross-checks these relationships against the dependencies declared by developers in the Puppet DSL. ProTI [Sokolowski and Salvaneschi 2023; Sokolowski et al. 2024] is used to test IaC programs written in Pulumi [Pulumi 2024], which is a provision-based IaC system.”
MethodVulnerabilities in infrastructure as code: what, how many, and who?
War, A., Diallo, A., Habib, A., Klein, J., & Bissyandé, T. F. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10672-8 · Cited by 25
Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety
Bilal, M., Crowcroft, J., Wang, R., Xu, X., & Dustdar, S. (2027) · Computer Science Review · doi:10.1016/j.cosrev.2026.101075 · Cited by 10
“A diagnosis often implies a change, but a change must be expressed as a structured proposal, checked outside the model, and applied with limited permissions [87], [88], [70], [68], [69].”
InfraFix: Technology-Agnostic Repair of Infrastructure as Code
Saavedra, N., Ferreira, J. F., & Mendes, A. S. (2025) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3713081.3731735 · Cited by 6
An empirical study of policy as code: adoption, purpose, and maintenance
Opdebeeck, R., Alfadel, M., Rahman, A., Kashiwa, Y., Ferreira, J. F., Kula, R. G., & De Roover, C. (2026) · Proceedings of the 23rd International Conference on Mining Software Repositories · doi:10.1145/3793302.3793355 · Cited by 5
“Researchers have studied several aspects of IaC development, including maintainability [9, 12, 30, 44, 47, 69, 71, 80], defect prediction [10, 11, 51, 59], testing [58, 74], and dependencies [41, 46].”
Metadata
- Authors
- Sokolowski, D., Spielmann, D., & Salvaneschi, G.
- Affiliations
- University of St.Gallen (CH)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2024-06
- Volume
- 50
- Issue
- 6
- Pages
- 1585-1599
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- 2326-3881
- OpenAlex
- W4396564044
- Cited by
- 20
- References
- 79
- Citations by year
- 2026: 12 · 2025: 5 · 2024: 3
- FWCI
- 20.51
- Type
- journal article
- Language
- en
- Open access
- Licence
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- Alexandria (UniSG) (University of St.Gallen) (submittedVersion, other-oa)
- Funders
- Swiss National Science Foundation (200429)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- infrastructure as code, Pulumi, TypeScript, test case generation, cloud infrastructure reliability, bug detection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.