The DOI matches its Crossref record; 43 of the 47 references it cites were found in a registry or an index.
Continuous Integration, Delivery and Deployment: A Systematic Review on Approaches, Tools, Challenges and Practices
Open accessAbstract
Continuous practices, i.e., continuous integration, delivery, and deployment, are the software development industry practices that enable organizations to frequently and reliably release new features and products. With the increasing interest in the literature on continuous practices, it is important to systematically review and synthesize the approaches, tools, challenges, and practices reported for adopting and implementing continuous practices. This paper aimed at systematically reviewing the state of the art of continuous practices to classify approaches and tools, identify challenges and practices in this regard, and identify the gaps for future research. We used the systematic literature review method for reviewing the peer-reviewed papers on continuous practices published between 2004 and June 1, 2016. We applied the thematic analysis method for analyzing the data extracted from reviewing 69 papers selected using predefined criteria. We have identified 30 approaches and associated tools, which facilitate the implementation of continuous practices in the following ways: (1) reducing build and test time in continuous integration (CI); (2) increasing visibility and awareness on build and test results in CI; (3) supporting (semi-) automated continuous testing; (4) detecting violations, flaws, and faults in CI; (5) addressing security and scalability issues in deployment pipeline; and (6) improving dependability and reliability of deployment process. We have also determined a list of critical factors, such as testing (effort and time), team awareness and transparency, good design principles, customer, highly skilled and motivated team, application domain, and appropriate infrastructure that should be carefully considered when introducing continuous practices in a given organization. The majority of the reviewed papers were validation (34.7%) and evaluation (36.2%) research types. This paper also reveals that continuous practices have been successfully applied to both greenfield and maintenance projects. Continuous practices have become an important area of software engineering research and practice. While the reported approaches, tools, and practices are addressing a wide range of challenges, there are several challenges and gaps, which require future research work for improving the capturing and reporting of contextual information in the studies reporting different aspects of continuous practices; gaining a deep understanding of how software-intensive systems should be (re-) architected to support continuous practices; and addressing the lack of knowledge and tools for engineering processes of designing and running secure deployment pipelines.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Despite the popularity of DevOps, security is among the major concerns that hinder the adoption of DevOps in software enterprises [5].”
From Prioritization based taxonomy of DevOps security challenges using PROMETHEE (2020) · cites this paper as [5]
In the citing paper
“We have also identified lack of automated testing tool performance measures for security [2], [5], [20], [23], [24] as a critical challenge that have a negative impact on DevOps security.”
From Prioritization based taxonomy of DevOps security challenges using PROMETHEE (2020) · cites this paper as [5]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
Systematic literature reviews in software engineering – A tertiary study
Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.03.006 · Cited by 1,230
Loveday Ph., D., Clement, E., & Phd, I. (2014) · doi:10.4337/9781783473069.00009 · Cited by 1,227
Requirements engineering paper classification and evaluation criteria: a proposal and a discussion
Wieringa, R., Maiden, N., Mead, N., & Rolland, C. (2006) · Requirements Engineering · doi:10.1007/s00766-005-0021-6 · Cited by 940
Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation
Humble, J., & Farley, D. R. (2010) · Cited by 911
Challenges and success factors for large-scale agile transformations: A systematic literature review
Dikert, K., Paasivaara, M., & Lassenius, C. (2016) · Journal of Systems and Software · doi:10.1016/j.jss.2016.06.013 · Cited by 890
Newman, S. (2015) · CERN Document Server (European Organization for Nuclear Research) · Cited by 768
Continuous software engineering: A roadmap and agenda
Fitzgerald, B. F., & Stol, K.-J. (2015) · Journal of Systems and Software · doi:10.1016/j.jss.2015.06.063 · Cited by 742
Identifying relevant studies in software engineering
Zhang, H., Babar, M. A., & Tell, P. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.12.010 · Cited by 570
Devops: A Software Architect's Perspective
Bass, L., Weber, I. M., & Zhu, L. (2015) · CERN Document Server (European Organization for Nuclear Research) · Cited by 561
Self-citation
Research works printed but not linked: 16
- 0, Longman Dictionary of Contemporary English Online
- luke, 2016, No One Agrees How to Define CI or CD
- 0, Camera Dictionary
- phillips, 2015, The IT Manager’s Guide to Continuous Delivery Delivering Business Value in Hours
- humble, 2016, Continuous delivery vs continuous deployment
- 2016, What is Continuous Deployment
- humble, 2016, What is Continuous Delivery?
- reed, 2016, The Business Case for Continuous Delivery
- skelton, 2016, Continuous Delivery With Windows and NET
- humble, 2015, Principle 2 Decouple Deployment and Release
- 2016, Five Habits of Highly Successful Continuous Delivery Practitioners
- thiele, 2016, Continuous Delivery An Easy Must-Have for Agile Development
- 2015, State of DevOps Report
- sundman, 2013, Continuous delivery vs continuous deployment
- suzie, 2016, The Product Managers Guide to Continuous Delivery and DevOps
- sato, 2015, Canary Release
Cited by
Sorted by citations
Items 101–200 of 678
Understanding, Debugging, and Optimizing Distributed Software Builds: A Design Study
Lebeuf, C., Voyloshnikova, E., Herzig, K., & Storey, M. (2018) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2018.00060 · Cited by 18
Lima, J. P., Mendonça, W. D. F., Vergílio, S. R., & Assunção, W. K. G. (2020) · International Software Product Line Conference (SPLC) · doi:10.1145/3382025.3414967 · Cited by 18
Advancing accountability in AI
OECD (2023) · OECD digital economy papers · doi:10.1787/2448f04b-en · Cited by 42
Robot learning on the job: Human-in-the-loop autonomy and learning during deployment
Liu, H., Nasiriany, S., Zhang, L., Bao, Z., & Zhu, Y. (2024) · The International Journal of Robotics Research · doi:10.1177/02783649241273901 · Cited by 23
A Continuous Certification Methodology for DevOps
Anisetti, M., Ardagna, C. A., Gaudenzi, F., & Damiani, E. (2019) · International Conference on Management of Emergent Digital EcoSystems (MEDES) · doi:10.1145/3297662.3365827 · Cited by 17
Integration of Security Standards in DevOps Pipelines: An Industry Case Study
Moyón, F., Soares, R., Pinto-Albuquerque, M., Mendez, D., & Beckers, K. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-64148-1_27 · Cited by 17
Towards a Framework for Continuous Software Engineering
Barcellos, M. P. (2020) · Brazilian Symposium on Software Engineering · doi:10.1145/3422392.3422469 · Cited by 17
Hannousse, A. (2021) · IET Software · doi:10.1049/sfw2.12011 · Cited by 22
The unified perspective of digital transformation and continuous software engineering
Klotins, E., & Peretz-Andersson, E. (2022) · doi:10.1145/3524614.3528626 · Cited by 17
Improving the Learning of Code Review Successive Tasks with Cross-Task Knowledge Distillation
Sghaier, O. B., & Sahraoui, H. A. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3643775 · Cited by 22
On the effects of continuous delivery on code quality: A case study in industry
Rubert, M., & Farias, K. (2021) · Computer Standards & Interfaces · doi:10.1016/j.csi.2021.103588 · Cited by 17
Continuous QoS-aware adaptation of Cloud-IoT application placements
Herrera, J. L., Berrocal, J., Forti, S., Antonio Brogi, A., & Murillo, J. M. (2023) · Computing · doi:10.1007/s00607-023-01153-1 · Cited by 16
Serverless Cloud Computing: State of the Art and Challenges
Lannurien, V., d’Orazio, L., Barais, O., & Boukhobza, J. (2023) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-3-031-26633-1_11 · Cited by 18
A Multi-Step Learning Approach to Assist Code Review
Sghaier, O. B., & Sahraoui, H. A. (2023) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner56733.2023.00049 · Cited by 17
How do Developers Talk about GitHub Actions? Evidence from Online Software Development Community
Zhang, Y., Wu, Y., Chen, T., Wang, T., Liu, H. P., & Wang, H. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3623327 · Cited by 16
Enabling continuous deployment techniques for quantum services
Romero‐Álvarez, J., Alvarado‐Valiente, J., Moguel, E., García-Alonso, J. M., & Murillo, J. M. (2024) · Software Practice and Experience · doi:10.1002/spe.3326 · Cited by 16
A Case for a New IT Ecosystem: On-The-Fly Computing
Karl, H., Kundisch, D., der Heide, F. M. A., & Wehrheim, H. (2019) · Business & Information Systems Engineering · doi:10.1007/s12599-019-00627-x · Cited by 19
How Useful is Code Change Information for Fault Localization in Continuous Integration?
Chen, A. R., Chen, T.-H. P., & Chen, J. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556931 · Cited by 16
An Empirical Study on Continuous Integration Trends, Topics and Challenges in Stack Overflow
Ouni, A., Saidani, I., AlOmar, E. A., & Mkaouer, M. W. (2023) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3593434.3593485 · Cited by 15
Continuous Intrusion: Characterizing the Security of Continuous Integration Services
Gu, Y., Ying, L., Chai, H., Qiao, C., Duan, H., & Gao, X. H. (2023) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp46215.2023.10179471 · Cited by 15
User Perspectives on Branching in Computer-Aided Design
Cheng, K., Cuvin, P., Olechowski, A., & Zhou, S. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610220 · Cited by 15
Systematic Literature Review on Application of Learning-Based Approaches in Continuous Integration
Arani, A. K., Le, T. H. M., Zahedi, M., & Babar, M. A. (2024) · IEEE Access · doi:10.1109/access.2024.3424276 · Cited by 15
Benjamin, J., & Mathew, J. (2024) · Computing · doi:10.1007/s00607-024-01370-2 · Cited by 15
On the Role of Software Architecture in DevOps Transformation
Shahin, M., & Babar, M. A. (2020) · International Conference on Software and System Process · doi:10.1145/3379177.3388891 · Cited by 15
Self-citationSuccess Factors when Transitioning to Continuous Deployment in Software-Intensive Embedded Systems
Dakkak, A., Mattos, D. I., & Bosch, J. (2021) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa53835.2021.00025 · Cited by 15
Tailored Blockchain Applications for the Natural Gas Industry: The Case Study of SOCAR
Zehir, C., Zehir, M., Borodin, A., Mamedov, Z. F., & Qurbanov, S. H. (2022) · Energies · doi:10.3390/en15166010 · Cited by 15
Improving problem identification via automated log clustering using dimensionality reduction
Rosenberg, C. M., & Moonen, L. (2018) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3239235.3239248 · Cited by 15
Dynamic adaptation of service-based applications: a design for adaptation approach
Sanctis, M. D., Bucchiarone, A., & Marconi, A. (2020) · Journal of Internet Services and Applications · doi:10.1186/s13174-020-00123-6 · Cited by 15
Continuous deployment in software-intensive system-of-systems
Dakkak, A., Bosch, J., Olsson, H. H., & Mattos, D. I. (2023) · Information and Software Technology · doi:10.1016/j.infsof.2023.107200 · Cited by 14
Nascimento, B. R., Santos, R., Henriques, J., Bernardo, M. V., & Caldeira, F. (2024) · Computers · doi:10.3390/computers13080192 · Cited by 14
Open Case Studies: Statistics and Data Science Education through Real-World Applications
Wright, C., Meng, Q., Breshock, M. R., Atta, L., Taub, M. A., Jager, L. R., Muschelli, J., & Hicks, S. C. (2024) · Journal of Statistics and Data Science Education · doi:10.1080/26939169.2024.2394541 · Cited by 14
Software test results exploration and visualization with continuous integration and nightly testing
Strandberg, P. E., Afzal, W., & Sundmark, D. (2022) · International Journal on Software Tools for Technology Transfer · doi:10.1007/s10009-022-00647-1 · Cited by 14
Kumar, A., Nadeem, M., & Shameem, M. (2023) · Automated Software Engineering · doi:10.1007/s10515-023-00388-8 · Cited by 16
Johanßen, J. O., Kleebaum, A., Paech, B., & Bruegge, B. (2019) · Journal of Software Evolution and Process · doi:10.1002/smr.2169 · Cited by 14
Software Testing in the DevOps Context: A Systematic Mapping Study
Pando, B., & Dávila, A. (2022) · Programming and Computer Software · doi:10.1134/s0361768822080175 · Cited by 14
IoTDeploy: Deployment of IoT Smart Applications over the Computing Continuum
Oliveira, F. B., Felice, M. D., & Kamienski, C. (2024) · Internet of Things · doi:10.1016/j.iot.2024.101348 · Cited by 14
An intelligent test management system for optimizing decision making during software testing
Lönnfält, A., Tu, V., Gay, G., Singh, A., & Tahvili, S. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112202 · Cited by 13
Continuous Reproducibility in GNSS Signal Processing
Fernández–Prades, C., Vilà‐Valls, J., Arribas, J., & Ramos, A. (2018) · IEEE Access · doi:10.1109/access.2018.2822835 · Cited by 12
The Dynamics of Software Composition AnalysisPreprint
Foo, D., Yeo, J., Xiao, H., & Sharma, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.00973 · Cited by 13
Machine Learning for Detecting Data Exfiltration: A ReviewPreprint
Sabir, B., Ullah, F., Babar, M. A., & Gaire, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.09344 · Cited by 13
Research Study on the Use of CI/CD Among Slovak Students
Hroncova, N., & Dakić, P. (2022) · International Conference on Advanced Computer Information Technologies (ACIT) · doi:10.1109/acit54803.2022.9913113 · Cited by 12
Ly, D., Overeem, M., Brinkkemper, S., & Dalpiaz, F. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112243 · Cited by 12
The Impact of AI Tools on Last-Mile Delivery in the E-Commerce Sector in Saudi Arabia
Ezmigna, I., Alghizzawi, M., alqsass, M. S., Abu-AlSondos, I. A., Abualfalayeh, G., & Aldulaimi, S. H. (2024) · International Conference on Decision Aid Sciences and Applications (DASA) · doi:10.1109/dasa63652.2024.10836527 · Cited by 12
Perceived benefits of Continuous Deployment in Software-Intensive Embedded Systems
Dakkak, A., Mattos, D. I., & Bosch, J. (2021) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac51774.2021.00126 · Cited by 12
An Introduction to Graphical Modeling of CI/CD Workflows with Rig
Tegeler, T., Teumert, S., Schürmann, J., Bainczyk, A., Busch, D., & Steffen, B. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-89159-6_1 · Cited by 12
Continuous Software Engineering in the Wild
Klotins, E., & Gorschek, T. (2022) · Lecture notes in business information processing · doi:10.1007/978-3-031-04115-0_1 · Cited by 12
Ononiwu, M., Azonuche, T. I., Imoh, P. O., & Enyejo, J. O. (2023) · International Journal of Scientific Research in Science and Technology · doi:10.32628/ijsrst2302542 · Cited by 12
Strategy for continuous testing in iDevOps
Zimmerer, P. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3183440.3183465 · Cited by 12
Foundational models and federated learning: survey, taxonomy, challenges and practical insights
Hatfaludi, C.-A., & Serban, A. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.2993 · Cited by 12
Why Do GitHub Actions Workflows Fail? An Empirical Study
Zheng, L., Li, S., Huang, X., Huang, J., Lin, B., Chen, J., & Xuan, J. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3749371 · Cited by 11
Analysis on the Use of Declarative and Pull-based Deployment Models on GitOps Using Argo CD
Ramadoni, Utami, E., & Fatta, H. A. (2021) · International Conference on Information and Communications Technology (ICOIACT) · doi:10.1109/icoiact53268.2021.9563984 · Cited by 11
The impact of a continuous integration service on the delivery time of merged pull requests
Bernardo, J. H., da Costa, D. A., Kulesza, U., & Treude, C. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10327-6 · Cited by 13
A Review on Continuous Integration and Continuous Deployment (CI/CD) for Machine Learning
Mahida, A. (2021) · International Journal of Science and Research (IJSR) · doi:10.21275/sr24314131827 · Cited by 11
Emerging Synergies: Industry 5.0 Integration of Complex Systems
Amirkhizi, P. J., Pedrammehr, S., Pakzad, S., Asady, Z., Arogbonlo, A., & Asadi, H. (2024) · IEEE International Systems Conference (SysCon) · doi:10.1109/syscon61195.2024.10553432 · Cited by 11
Implementing Continuous Integration and Continuous Deployment (CI/CD) in Modern Software Development
Jani, Y. (2023) · International Journal of Science and Research (IJSR) · doi:10.21275/sr24716120535 · Cited by 11
Quantitative DevSecOps Metrics for Cloud-Based Web Microservices
Zhang, J. Y., & Zhang, Y. (2024) · IEEE Access · doi:10.1109/access.2024.3486314 · Cited by 11
Empirical Investigation of Key Enablers for Secure DevOps Practices
Akbar, M. A., & Alsanad, A. A. (2025) · IEEE Access · doi:10.1109/access.2025.3549183 · Cited by 11
Debroy, V., & Miller, S. (2019) · IEEE Software · doi:10.1109/ms.2019.2947004 · Cited by 11
DevOps Model Appproach for Monitoring Smart Energy Systems
Lévy, L.-N., Bosom, J., Guérard, G., Amor, S. B., Bui, M., & Tran, H. (2022) · Energies · doi:10.3390/en15155516 · Cited by 11
DupHunter: Detecting Duplicate Pull Requests in Fork-Based Development
Jiang, H., Li, Y., Guo, S., Li, X., Zhang, T., Li, H., & Chen, R. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3235942 · Cited by 11
Experiment: Sizing Exposed Credentials in GitHub Public Repositories for CI/CD
Yaşar, H. (2018) · IEEE Cybersecurity Development (SecDev) · doi:10.1109/secdev.2018.00039 · Cited by 11
Amalfitano, D., Simone, V. D., Maietta, R. R., Scala, S., & Fasolino, A. R. (2019) · Journal of Software Evolution and Process · doi:10.1002/smr.2171 · Cited by 11
A Model-driven Approach to Continuous Practices for Modern Cloud-based Web Applications
Tegeler, T., Gossen, F., & Steffen, B. (2019) · International Conference on Cloud Computing, Data Science & Engineering (Confluence) · doi:10.1109/confluence.2019.8776962 · Cited by 11
When Security Meets Velocity: Modeling Continuous Security for Cloud Applications using DevSecOps
Kumar, R., & Goyal, R. (2021) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-981-15-9651-3_36 · Cited by 11
Adaptive Reward Computation in Reinforcement Learning-Based Continuous Integration Testing
Yang, Y., Pan, C., Li, Z., & Zhao, R. (2021) · IEEE Access · doi:10.1109/access.2021.3063232 · Cited by 11
Closing the Feedback Loop in DevOps Through Autonomous Monitors in Operations
Hrusto, A., Runeson, P., & Engström, E. (2021) · SN Computer Science · doi:10.1007/s42979-021-00826-y · Cited by 11
Krauss, P., Touré, V., Gnodtke, K., Crameri, K., & Oesterle, S. (2021) · Applied Sciences · doi:10.3390/app112311311 · Cited by 11
A Literature Review on the Challenges of Applying Test-Driven Development in Software Engineering
Staegemann, D., Volk, M., Perera, M., Haertel, C., Pohl, M., Daase, C., & Turowski, K. (2022) · Complex Systems Informatics and Modeling Quarterly · doi:10.7250/csimq.2022-31.02 · Cited by 10
Good modelling software practices
Lemmen, C., & Sommer, P. S. (2024) · Ecological Modelling · doi:10.1016/j.ecolmodel.2024.110890 · Cited by 10
From automation to CI/CD: a comparative evaluation of threat modeling tools
Landuyt, D. V., Sion, L., Philips, W., & Joosen, W. (2024) · IEEE Secure Development Conference (SecDev) · doi:10.1109/secdev61143.2024.00010 · Cited by 10
Feoktistov, A. G., Gorsky, S. A., Sidorov, I., Bychkov, I. V., Tchernykh, A. N., & Еделев, А. В. (2020) · Communications in computer and information science · doi:10.1007/978-3-030-41005-6_2 · Cited by 10
Hybrid model for evaluation of quality aware DevOps
Batra, P., & Jatain, A. (2021) · International Journal of Applied Science and Engineering · doi:10.6703/ijase.202106_18(2).013 · Cited by 10
Incorporating energy efficiency measurement into CI\CD pipeline
Kruglov, A. V., Succi, G., & Vasuez, X. (2021) · European Symposium on Software Engineering (ESSE) · doi:10.1145/3501774.3501777 · Cited by 10
Continuous and Secure Integration Framework for Smart Contracts
Reyes, Á., Jimeno, M., & Villanueva-Polanco, R. (2023) · Sensors · doi:10.3390/s23010541 · Cited by 10
Bali, M. K., & Walia, R. (2023) · International Conference on Computing, Communication, and Intelligent Systems (ICCCIS) · doi:10.1109/icccis60361.2023.10425162 · Cited by 10
A new metamodel approach of CI/CD applied to Internet of Things Ecosystem
Khalyly, B. E., Belangour, A., Banane, M., & Erraissi, A. (2020) · IEEE International Conference on Electronics, Control, Optimization and Computer Science (ICECOCS) · doi:10.1109/icecocs50124.2020.9314485 · Cited by 10
de Almeida, C. D. A., Feijó, D. N., & Rocha, L. S. (2022) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3524842.3528049 · Cited by 10
Build Optimization: A Systematic Literature Review
Aïdasso, H., Sayagh, M., & Bordeleau, F. (2025) · ACM Computing Surveys · doi:10.1145/3757912 · Cited by 9
Towards semantic versioning of open pre-trained language model releases on hugging face
Ajibode, A., Bangash, A. A., Cogo, F. R., Adams, B., & Hassan, A. E. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10631-3 · Cited by 19
A Multi-Agent LLM Environment for Software Design and Refactoring: A Conceptual Framework
Rajendran, V., Besiahgari, D., Patil, S. C., Chandrashekaraiah, M., & Challagulla, V. (2025) · IEEE SoutheastCon · doi:10.1109/southeastcon56624.2025.10971563 · Cited by 9
A Survey on Immersive Cyber Situational Awareness Systems
Ahmad, H., Ullah, F., & Jafri, R. (2025) · Journal of Cybersecurity and Privacy · doi:10.3390/jcp5020033 · Cited by 9
Aligned, Purpose-Driven Cooperation: The Future Way of System Development
Zweihoff, P., Tegeler, T., Schürmann, J., Bainczyk, A., & Steffen, B. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-89159-6_27 · Cited by 13
StalkCD: A Model-Driven Framework for Interoperability and Analysis of CI/CD Pipelines
Düllmann, T. F., Kabierschke, O., & van Hoorn, A. (2021) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa53835.2021.00035 · Cited by 9
Huber, F., & Hagel, G. (2022) · International Convention on Information and Communication Technology, Electronics and Microelectronics · doi:10.23919/mipro55190.2022.9803560 · Cited by 9
A Microservice Deployment Guide
Nino-Martinez, V. M., Ocharán-Hernández, J. O., Limón, X., & Pérez-Arriaga, J. C. (2022) · Programming and Computer Software · doi:10.1134/s0361768822080151 · Cited by 9
Analysing the Analysers: An Investigation of Source Code Analysis Tools
Bhutani, V., Toosi, F. G., & Buckley, J. (2024) · Applied Computer Systems · doi:10.2478/acss-2024-0013 · Cited by 9
Stich, V., Hoffmann, J., & Heimes, P. (2017) · HMD Praxis der Wirtschaftsinformatik · doi:10.1365/s40702-017-0386-2 · Cited by 9
Researching cooperation and communication in continuous software engineering
Dittrich, Y., Jacob Nørbjerg, B., Tell, P., & Bendix, L. (2018) · International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE) · doi:10.1145/3195836.3195856 · Cited by 9
Running Simulations in HPC and Cloud Resources by Implementing Enhanced TOSCA Workflows
Carnero, J., & Nieto, F. J. (2018) · International Conference on High Performance Computing & Simulation (HPCS) · doi:10.1109/hpcs.2018.00075 · Cited by 9
Efficient and Flexible Test Automation in Production Systems Engineering
Winkler, D., Meixner, K., & Novák, P. (2019) · Security and Quality in Cyber-Physical Systems Engineering · doi:10.1007/978-3-030-25312-7_9 · Cited by 5
Kostromin, R. O. (2020) · International Workshop on Information, Computation, and Control Systems for Distributed Environments (ICCS-DE) · doi:10.47350/iccs-de.2020.15 · Cited by 9
Zhang, J., Ferydouni, R., Montana, A., Bittman, D., & Alvaro, P. (2021) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/3472883.3486986 · Cited by 9
What is There About DevOps Assessment? A Systematic Mapping
Orozco-Garcés, C.-E., Pardo, C., & Salazar-Mondragón, Y.-H. (2022) · Revista Facultad de Ingeniería · doi:10.19053/01211129.v31.n59.2022.13896 · Cited by 9
Agent-Based DevOps of Software and Hardware Resources for Digital Twins of Infrastructural Objects
Kostromin, R. O., & Feoktistov, A. G. (2020) · International Conference on Future Networks and Distributed Systems (ICFNDS) · doi:10.1145/3440749.3442599 · Cited by 10
Automated System-Level Software Testing of Industrial Networked Embedded SystemsPreprint
Strandberg, P. E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.08312 · Cited by 9
Understanding the quality and evolution of Android app build systems
Liu, P., Li, L., Liu, K., McIntosh, S., & Grundy, J. (2023) · Journal of Software Evolution and Process · doi:10.1002/smr.2602 · Cited by 8
Automated NFR testing in continuous integration environments: a multi-case study of Nordic companies
Yu, L., Alégroth, E., Chatzipetrou, P., & Gorschek, T. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10356-1 · Cited by 8
Challenges Affecting the Successful Adoption of DevOps Practices: A Systematic Literature Review
Nayanajith, A., & Wickramarachchi, R. (2024) · International Conference on Advanced Research in Computing (ICARC) · doi:10.1109/icarc61713.2024.10499735 · Cited by 12
Plotnikov, A. V., Demiryürek, K., Plotnikova, A., ANDREEVA, O. Y., & Suzdaleva, G. (2024) · Lecture notes in information systems and organisation · doi:10.1007/978-3-031-66801-2_10 · Cited by 8
Petrakis, K., Agorogiannis, E., Antonopoulos, G., Anagnostopoulos, T., Grigoropoulos, N., Veroni, E., Berne, A., Azaiez, S., Benomar, Z., Kakoulidis, H., et al. (2025) · Software · doi:10.3390/software4020010 · Cited by 8
Metadata
- Authors
- Shahin, M., Ali Babar, M., & Zhu, L.
- Affiliations
- The University of Adelaide (AU)
- Commonwealth Scientific and Industrial Research Organisation (AU)
- Venue
- IEEE Access
- Published
- 2017
- Volume
- 5
- Pages
- 3909-3943
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W2603119212
- Cited by
- 680
- References
- 47
- Citations by year
- 2026: 86 · 2025: 111 · 2024: 89 · 2023: 97 · 2022: 89 · 2021: 83 · 2020: 49 · 2019: 38 · 2018: 33 · 2017: 5
- FWCI
- 10.80
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion)
- arXiv (Cornell University) (acceptedVersion)
- RMIT Research Repository (RMIT University Library) (acceptedVersion)
- Adelaide Research & Scholarship (AR&S) (University of Adelaide) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- arXiv (Cornell University)
- Funders
- Data61, a business unit of CSIRO, Australia
- Australian Government Research Training Program Scholarship
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- continuous integration, continuous delivery, continuous deployment, continuous practice, deployment pipelines, scalability, barriers to adoption
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
