The DOI matches its Crossref record; 43 of the 47 references it cites were found in a registry or an index.
Continuous Integration, Delivery and Deployment: A Systematic Review on Approaches, Tools, Challenges and Practices
Open accessAbstract
Continuous practices, i.e., continuous integration, delivery, and deployment, are the software development industry practices that enable organizations to frequently and reliably release new features and products. With the increasing interest in the literature on continuous practices, it is important to systematically review and synthesize the approaches, tools, challenges, and practices reported for adopting and implementing continuous practices. This paper aimed at systematically reviewing the state of the art of continuous practices to classify approaches and tools, identify challenges and practices in this regard, and identify the gaps for future research. We used the systematic literature review method for reviewing the peer-reviewed papers on continuous practices published between 2004 and June 1, 2016. We applied the thematic analysis method for analyzing the data extracted from reviewing 69 papers selected using predefined criteria. We have identified 30 approaches and associated tools, which facilitate the implementation of continuous practices in the following ways: (1) reducing build and test time in continuous integration (CI); (2) increasing visibility and awareness on build and test results in CI; (3) supporting (semi-) automated continuous testing; (4) detecting violations, flaws, and faults in CI; (5) addressing security and scalability issues in deployment pipeline; and (6) improving dependability and reliability of deployment process. We have also determined a list of critical factors, such as testing (effort and time), team awareness and transparency, good design principles, customer, highly skilled and motivated team, application domain, and appropriate infrastructure that should be carefully considered when introducing continuous practices in a given organization. The majority of the reviewed papers were validation (34.7%) and evaluation (36.2%) research types. This paper also reveals that continuous practices have been successfully applied to both greenfield and maintenance projects. Continuous practices have become an important area of software engineering research and practice. While the reported approaches, tools, and practices are addressing a wide range of challenges, there are several challenges and gaps, which require future research work for improving the capturing and reporting of contextual information in the studies reporting different aspects of continuous practices; gaining a deep understanding of how software-intensive systems should be (re-) architected to support continuous practices; and addressing the lack of knowledge and tools for engineering processes of designing and running secure deployment pipelines.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Despite the popularity of DevOps, security is among the major concerns that hinder the adoption of DevOps in software enterprises [5].”
From Prioritization based taxonomy of DevOps security challenges using PROMETHEE (2020) · cites this paper as [5]
In the citing paper
“We have also identified lack of automated testing tool performance measures for security [2], [5], [20], [23], [24] as a critical challenge that have a negative impact on DevOps security.”
From Prioritization based taxonomy of DevOps security challenges using PROMETHEE (2020) · cites this paper as [5]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
Systematic literature reviews in software engineering – A tertiary study
Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M., & Linkman, S. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.03.006 · Cited by 1,230
Loveday Ph., D., Clement, E., & Phd, I. (2014) · doi:10.4337/9781783473069.00009 · Cited by 1,227
Requirements engineering paper classification and evaluation criteria: a proposal and a discussion
Wieringa, R., Maiden, N., Mead, N., & Rolland, C. (2006) · Requirements Engineering · doi:10.1007/s00766-005-0021-6 · Cited by 940
Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation
Humble, J., & Farley, D. R. (2010) · Cited by 911
Challenges and success factors for large-scale agile transformations: A systematic literature review
Dikert, K., Paasivaara, M., & Lassenius, C. (2016) · Journal of Systems and Software · doi:10.1016/j.jss.2016.06.013 · Cited by 890
Newman, S. (2015) · CERN Document Server (European Organization for Nuclear Research) · Cited by 768
Continuous software engineering: A roadmap and agenda
Fitzgerald, B. F., & Stol, K.-J. (2015) · Journal of Systems and Software · doi:10.1016/j.jss.2015.06.063 · Cited by 742
Identifying relevant studies in software engineering
Zhang, H., Babar, M. A., & Tell, P. (2010) · Information and Software Technology · doi:10.1016/j.infsof.2010.12.010 · Cited by 570
Devops: A Software Architect's Perspective
Bass, L., Weber, I. M., & Zhu, L. (2015) · CERN Document Server (European Organization for Nuclear Research) · Cited by 561
Self-citation
Flagged first, then by citations
4 of the 47 references couldn’t be matched to a known paper.
Research works printed but not linked: 16
- 0, Longman Dictionary of Contemporary English Online
- luke, 2016, No One Agrees How to Define CI or CD
- 0, Camera Dictionary
- phillips, 2015, The IT Manager’s Guide to Continuous Delivery Delivering Business Value in Hours
- humble, 2016, Continuous delivery vs continuous deployment
- 2016, What is Continuous Deployment
- humble, 2016, What is Continuous Delivery?
- reed, 2016, The Business Case for Continuous Delivery
- skelton, 2016, Continuous Delivery With Windows and NET
- humble, 2015, Principle 2 Decouple Deployment and Release
- 2016, Five Habits of Highly Successful Continuous Delivery Practitioners
- thiele, 2016, Continuous Delivery An Easy Must-Have for Agile Development
- 2015, State of DevOps Report
- sundman, 2013, Continuous delivery vs continuous deployment
- suzie, 2016, The Product Managers Guide to Continuous Delivery and DevOps
- sato, 2015, Canary Release
Cited by
Sorted by citations
Economics of artificial intelligence in healthcare: Diagnosis vs. treatment
Khanna, N. N., Maindarkar, M. A., Viswanathan, V., Fernandes, J. F. E., Paul, S., Bhagawati, M., Ahluwalia, P., Ruzsa, Z., Sharma, A., Kolluri, R., et al. (2022) · Healthcare · doi:10.3390/healthcare10122493 · Cited by 347
Challenges and solutions when adopting DevSecOps: A systematic review
Rajapakse, R. N., Zahedi, M., Babar, M. A., & Shen, H. (2022) · Information and Software Technology · doi:10.1016/j.infsof.2021.106700 · Cited by 235
DevOps and software quality: A systematic mapping
Mishra, A., & Otaiwi, Z. (2020) · Computer Science Review · doi:10.1016/j.cosrev.2020.100308 · Cited by 177
A Survey on Industrial Information Integration 2016–2019
Chen, Y. (2019) · Journal of Industrial Integration and Management · doi:10.1142/s2424862219500167 · Cited by 124
Dissanayake, N., Jayatilaka, A., Zahedi, M., & Babar, M. A. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106771 · Cited by 119
Challenges With Developing Secure Mobile Health Applications: Systematic Review
Aljedaani, B., & Babar, M. A. (2021) · JMIR mhealth and uhealth · doi:10.2196/15654 · Cited by 114
Test Case Prioritization in Continuous Integration environments: A systematic mapping study
Lima, J. P., & Vergílio, S. R. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106268 · Cited by 99
Arachchi, S. A. I. B. S., & Perera, I. (2018) · 2018 Moratuwa Engineering Research Conference (MERCon) · doi:10.1109/mercon.2018.8421965 · Cited by 97
The Application of Blockchain in Social Media: A Systematic Literature Review
Hisseine, M. A., Chen, D., & Yang, X. (2022) · Applied Sciences · doi:10.3390/app12136567 · Cited by 99
Kumar, R., & Goyal, R. (2020) · Computers & Security · doi:10.1016/j.cose.2020.101967 · Cited by 97
Singh, V., Kumar, V. S., & Singh, V. B. (2022) · Decision Analytics Journal · doi:10.1016/j.dajour.2022.100159 · Cited by 82
An Empirical Study of Investigating Mobile Applications Development Challenges
Ahmad, A., Li, K., Feng, C., Asim, S. M., Yousif, A., & Shi, G. (2018) · IEEE Access · doi:10.1109/access.2018.2818724 · Cited by 79
We’re doing it live: A multi-method empirical study on continuous experimentation
Schermann, G., Cito, J., Leitner, P., Zdun, U., & Gall, H. C. (2018) · Information and Software Technology · doi:10.1016/j.infsof.2018.02.010 · Cited by 74
From Ad-Hoc Data Analytics to DataOps
Munappy, A. R., Mattos, D. I., Bosch, J., Olsson, H. H., & Dakkak, A. (2020) · International Conference on Software and System Process · doi:10.1145/3379177.3388909 · Cited by 73
From DevOps to MLOps: Overview and Application to Electricity Market Forecasting
Subramanya, R., Sierla, S., & Vyatkin, V. V. (2022) · Applied Sciences · doi:10.3390/app12199851 · Cited by 79
An empirical study of architecting for continuous delivery and deployment
Shahin, M., Zahedi, M., Babar, M. A., & Zhu, L. (2018) · Empirical Software Engineering · doi:10.1007/s10664-018-9651-4 · Cited by 68
Self-citationUncovering the Benefits and Challenges of Continuous Integration Practices
Elazhary, O., Werner, C., Li, Z. S., Lowlind, D., Ernst, N., & Storey, M. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3064953 · Cited by 67
Barbie, A., Pech, N., Hasselbring, W., Flogel, S., Wenzhöfer, F., Walter, M., Shchekinova, E., Busse, M., Turk, M., Hofbauer, M., et al. (2021) · IEEE Internet Computing · doi:10.1109/mic.2021.3065245 · Cited by 65
Beyond Continuous Delivery: An Empirical Investigation of Continuous Deployment Challenges
Shahin, M., Babar, M. A., Zahedi, M., & Zhu, L. (2017) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2017.18 · Cited by 64
Self-citationToward the Observability of Cloud-Native Applications: The Overview of the State-of-the-Art
Kosińska, J., Baliś, B., Konieczny, M., Malawski, M., & Zieliński, S. (2023) · IEEE Access · doi:10.1109/access.2023.3281860 · Cited by 63
Machine Learning for Detecting Data Exfiltration
Sabir, B., Ullah, F., Babar, M. A., & Gaire, R. (2021) · ACM Computing Surveys · doi:10.1145/3442181 · Cited by 78
An Empirical Study on Culture, Automation, Measurement, and Sharing of DevSecOps
Tomas, N., Li, J., & Huang, H. (2019) · 2019 International Conference on Cyber Security and Protection of Digital Services (Cyber Security) · doi:10.1109/cybersecpods.2019.8884935 · Cited by 66
Donca, I.-C., Stan, O. P., Misaroş, M., Gota, D.-I., & Miclea, L. C. (2022) · Sensors · doi:10.3390/s22124637 · Cited by 60
Zhang, J., & Li, J. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106296 · Cited by 66
Designing and constructing internet-of-Things systems: An overview of the ecosystem
Dias, J. P., Restivo, A., & Ferreira, H. S. (2022) · Internet of Things · doi:10.1016/j.iot.2022.100529 · Cited by 63
On the usage, co-usage and migration of CI/CD tools: A qualitative analysis
Mazrae, P. R., Mens, T., Golzadeh, M., & Decan, A. (2023) · Empirical Software Engineering · doi:10.1007/s10664-022-10285-5 · Cited by 58
Sarker, O., Jayatilaka, A., Haggag, S., Liu, C., & Babar, M. A. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111899 · Cited by 62
NLP methods in host-based intrusion detection systems: A systematic review and future directions
Sworna, Z. T., Mousavi, Z., & Babar, M. A. (2023) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2023.103761 · Cited by 73
A survey of the state of the practice for research software in the United States
Carver, J. C., Weber, N., Ram, K., Gesing, S., & Katz, D. S. (2022) · PeerJ Computer Science · doi:10.7717/peerj-cs.963 · Cited by 55
One size does not fit all: an empirical study of containerized continuous deployment workflows
Zhang, Y., Vasilescu, B., Wang, H., & Filkov, V. (2018) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3236024.3236033 · Cited by 52
Prioritization based taxonomy of DevOps security challenges using PROMETHEE
Rafi, S., Yu, W., Akbar, M. A., Alsanad, A., & Gumaei, A. (2020) · IEEE Access · doi:10.1109/access.2020.2998819 · Cited by 50
Metaverse healthcare supply chain: Conceptual framework and barrier identification
Chen, Z., & Ruan, J.-Q. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108113 · Cited by 56
The effects of continuous integration on software development: a systematic literature review
Soares, E., Sizílio, G., Santos, J., da Costa, D. A., & Kulesza, U. (2022) · Empirical Software Engineering · doi:10.1007/s10664-021-10114-1 · Cited by 54
Capabilities and Practices in DevOps: A Multivocal Literature Review
Amaro, R., Pereira, R., & da Silva, M. M. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3166626 · Cited by 43
Towards Autonomous Testing Agents via Conversational Large Language Models
Feldt, R., Kang, S., Yoon, J., & Yoo, S. (2023) · 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00148 · Cited by 49
Controlled experimentation in continuous experimentation: Knowledge and challenges
Auer, F., Ros, R., Kaltenbrunner, L., Runeson, P., & Felderer, M. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106551 · Cited by 4
Establishing multilevel test-to-code traceability links
White, R., Krinke, J., & Tan, R. R. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380921 · Cited by 41
How do Practitioners Capture and Utilize User Feedback During Continuous Software Engineering?
Johanßen, J. O., Kleebaum, A., Bruegge, B., & Paech, B. (2019) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re.2019.00026 · Cited by 39
Lombardi, F., & Fanton, A. (2023) · Software Quality Journal · doi:10.1007/s11219-023-09619-3 · Cited by 39
Adapting Software Architectures to Machine Learning Challenges
Serban, A., & Visser, J. (2022) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner53432.2022.00029 · Cited by 49
Modern Code Reviews—Survey of Literature and Practice
Badampudi, D., Unterkalmsteiner, M., & Britto, R. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3585004 · Cited by 37
Filippou, K., Aifantis, G., Papakostas, G. A., & Tsekouras, G. E. (2023) · Information · doi:10.3390/info14040232 · Cited by 35
Trovato, M., Belluomo, L., Bici, M., Prist, M., Campana, F., & Cicconi, P. (2025) · The International Journal of Advanced Manufacturing Technology · doi:10.1007/s00170-025-15273-9 · Cited by 35
Efficient static checking of library updates
Foo, D., Chua, H., Yeo, J., Ang, M. Y., & Sharma, A. (2018) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3236024.3275535 · Cited by 35
Ammad, G., Janjua, U. I., Madni, T. M., Cheema, M. F., & Shahid, A. R. (2019) · IEEE Access · doi:10.1109/access.2019.2953008 · Cited by 35
A maturity model for secure requirements engineering
Niazi, M., Saeed, A. M., Alshayeb, M., Mahmood, S., & Zafar, S. N. (2020) · Computers & Security · doi:10.1016/j.cose.2020.101852 · Cited by 35
Redundancy, Context, and Preference: An Empirical Study of Duplicate Pull Requests in OSS Projects
Li, Z., Yu, Y., Zhou, M., Wang, T., Yin, G., Lan, L., & Wang, H. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.3018726 · Cited by 38
Software Engineering Applications Enabled by Blockchain Technology: A Systematic Mapping Study
Demi, S., Colomo‐Palacios, R., & Sánchez‐Gordón, M. (2021) · Applied Sciences · doi:10.3390/app11072960 · Cited by 36
Wang, Y., Mäntylä, M., Liu, Z., & Markkula, J. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111259 · Cited by 34
Machine learning operations landscape: platforms and tools
Berberi, L., Kozlov, V. Y., Nguyen, G., Díaz, J. S.-P., Calatrava, A., Moltó, G., Tran, V., & García, Á. L. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11164-3 · Cited by 47
Asif, M., & Ahmed, J. (2020) · IEEE Access · doi:10.1109/access.2020.2999036 · Cited by 33
Challenges in Migration from Waterfall to Agile Environments
Almeida, F. L. (2017) · World Journal of Computer Application and Technology · doi:10.13189/wjcat.2017.050302 · Cited by 33
Strandberg, P. E., Enoiu, E. P., Afzal, W., Sundmark, D., & FELDT, R. H. (2019) · IEEE Access · doi:10.1109/access.2019.2909093 · Cited by 31
Grande, R., Vizcaíno, A., & García, F. O. (2024) · Computer Standards & Interfaces · doi:10.1016/j.csi.2023.103767 · Cited by 30
The Role of DevOps in Streamlining Software Delivery: Key Practices for Seamless CI/CD
Modalavalasa, G. (2021) · International Journal of Advanced Research in Science Communication and Technology · doi:10.48175/ijarsct-8978c · Cited by 30
Using Autoencoders for Data-Driven Analysis in Internal Auditing
Nonnenmacher, J., Kruse, F., Schumann, G., & Gómez, J. M. (2021) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2021.697 · Cited by 30
Automating Root Cause Analysis via Machine Learning in Agile Software Testing Environments
Kahles, J., Torronen, J., Huuhtanen, T., & Jung, A. (2019) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2019.00047 · Cited by 29
Mining Questions Asked about Continuous Software Engineering
Zahedi, M., Rajapakse, R. N., & Babar, M. A. (2020) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3383219.3383224 · Cited by 29
On Threats to the 5G Service Based Architecture
Køien, G. M. (2021) · Wireless Personal Communications · doi:10.1007/s11277-021-08200-0 · Cited by 31
Towards Continuous Deployment for Blockchain
Górski, T. (2021) · Applied Sciences · doi:10.3390/app112411745 · Cited by 29
Towards cost-benefit evaluation for continuous software engineering activities
Klotins, E., Gorschek, T., Sundelin, K., & Falk, E. A. (2022) · Empirical Software Engineering · doi:10.1007/s10664-022-10191-w · Cited by 29
Helal, K. M., Fragasso, J., & Moro, L. (2024) · Ocean Engineering · doi:10.1016/j.oceaneng.2024.116993 · Cited by 28
When life gives you oranges: detecting and diagnosing intermittent job failures at Mozilla
Lampel, J., Just, S., Apel, S., & Zeller, A. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3473931 · Cited by 28
SLA Driven Program Planning for Cloud Delivery Pipelines
Nagalla, S., & Remella, S. (2026) · International Conference on Smart Futuristic Technology (SMARTTECH) · doi:10.1109/icsft66733.2026.11507012 · Cited by 27
Let's Supercharge the Workflows: An Empirical Study of GitHub Actions
Chen, T., Zhang, Y., Chen, S., Wang, T., & Wu, Y. (2021) · IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) · doi:10.1109/qrs-c55045.2021.00163 · Cited by 27
Destructive Criticism in Software Code Review Impacts Inclusion
Gunawardena, S. D., Devine, P., Beaumont, I., Garden, L. P., Murphy-Hill, E. R., & Blincoe, K. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555183 · Cited by 25
Ambush From All Sides: Understanding Security Threats in Open-Source Software CI/CD Pipelines
Pan, Z., Shen, W., Wang, X., Yang, Y., Chang, R., Liu, Y., Liu, C., Liu, Y., & Ren, K. (2023) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2023.3253572 · Cited by 27
Summarization techniques for code, change, testing, and user feedback (Invited paper)
Panichella, S. (2018) · 2018 IEEE Workshop on Validation, Analysis and Evolution of Software Tests (VST) · doi:10.1109/vst.2018.8327148 · Cited by 26
Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deployment
Liu, H., Nasiriany, S., Zhang, L., Bao, Z., & Zhu, Y. (2023) · Robotics: Science and Systems · doi:10.15607/rss.2023.xix.005 · Cited by 53
Software engineering practices for machine learning — Adoption, effects, and team assessment
Serban, A., van der Blom, K., Hoos, H. H., & Visser, J. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111907 · Cited by 34
Assessing test artifact quality—A tertiary study
Tran, H. K. V., Unterkalmsteiner, M., Börstler, J., & bin Ali, N. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106620 · Cited by 26
On the outdatedness of workflows in the GitHub Actions ecosystem
Decan, A., Mens, T., & Delicheh, H. O. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111827 · Cited by 24
Ravindran, A. (2023) · IoT · doi:10.3390/iot4040021 · Cited by 34
Soenen, T., Rossem, S. V., Tavernier, W., Vicens, F., Valocchi, D., Trakadas, P., Karkazis, P., Xilouris, G. K., Eardley, P. L., Kolometsos, S., et al. (2018) · IEEE/IFIP Network Operations and Management Symposium (NOMS) · doi:10.1109/noms.2018.8406139 · Cited by 24
Intermittently failing tests in the embedded systems domain
Strandberg, P. E., Ostrand, T. J., Weyuker, E. J., Afzal, W., & Sundmark, D. (2020) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3395363.3397359 · Cited by 24
A Comprehensive Bibliometric Assessment on Software Testing (2016–2021)
Zardari, S., Alam, S., Abosaq, H. A., Reshan, M. S. A., Shaikh, A., Malik, A. F. K., Rehman, M. M. U., & Mouratidis, H. (2022) · Electronics · doi:10.3390/electronics11131984 · Cited by 24
Software for Improve the Security of Kubernetes-based CI/CD Pipeline
Shevchuk, R., Karpiński, M., Kasianchuk, M. M., Yakymenko, I., Melnyk, A., & Tykhyi, R. (2023) · International Conference on Advanced Computer Information Technologies (ACIT) · doi:10.1109/acit58437.2023.10275654 · Cited by 23
Architectural Design Space for Modelling and Simulation as a Service: A Review
Shahin, M., Babar, M. A., & Chauhan, M. A. (2020) · Journal of Systems and Software · doi:10.1016/j.jss.2020.110752 · Cited by 23
Self-citationAn Advanced DevOps Environment for Microservice-based Applications
Throner, S., Hutter, H., Sänger, N., Schneider, M., Hanselmann, S., Petrovic, P., & Abeck, S. (2021) · IEEE International Symposium on Service-Oriented System Engineering (SOSE) · doi:10.1109/sose52839.2021.00020 · Cited by 24
Practitioners' eye on continuous software engineering
Johanßen, J. O., Kleebaum, A., Paech, B., & Bruegge, B. (2018) · International Conference on Software and System Process · doi:10.1145/3202710.3203150 · Cited by 21
FlakyCat: Predicting Flaky Tests Categories using Few-Shot Learning
Akli, A., Haben, G., Habchi, S., Papadakis, M., & Traon, Y. L. (2023) · doi:10.1109/ast58925.2023.00018 · Cited by 35
AI-Driven Refactoring: A Pipeline for Identifying and Correcting Data Clumps in Git Repositories
Baumgartner, N., Iyenghar, P., Schoemaker, T., & Pulvermüller, E. (2024) · Electronics · doi:10.3390/electronics13091644 · Cited by 27
Pope, N., Kahila, J., Vartiainen, H., & Tedre, M. (2025) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2025.3529994 · Cited by 29
Qiu, S., Zaheer, Q., Shah, S. M. A. H., Shah, S. F. H., Ehsan, H., Atta, Z., Ai, C., Wang, J., Wang, W., & Peng, J. (2024) · Measurement · doi:10.1016/j.measurement.2024.116557 · Cited by 20
Testing and deployment patterns for the internet-of-things
Dias, J. P., Ferreira, H. S., & Sousa, T. B. (2019) · European Conference on Pattern Languages of Programs · doi:10.1145/3361149.3361165 · Cited by 21
Rosenberg, C. M., & Moonen, L. (2020) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3382494.3410684 · Cited by 22
Arbas, S. M., Busi, S. B., Queirós, P., de Nies, L., Herold, M., May, P., Wilmes, P., Muller, E. E. L., & Narayanasamy, S. (2021) · Frontiers in Genetics · doi:10.3389/fgene.2021.666244 · Cited by 25
A qualitative study of architectural design issues in DevOps
Shahin, M., Nasab, A. R., & Babar, M. A. (2021) · Journal of Software Evolution and Process · doi:10.1002/smr.2379 · Cited by 20
Self-citationDevOps adoption: Insights from a large European Telco
Trigo, A., Varajão, J. E., & Sousa, L. (2022) · Cogent Engineering · doi:10.1080/23311916.2022.2083474 · Cited by 24
The k + 1 Symmetric Test Pattern for Smart Contracts
Górski, T. (2022) · Symmetry · doi:10.3390/sym14081686 · Cited by 20
Salam, M. A., & Khan, S. U. (2018) · Journal of Software Evolution and Process · doi:10.1002/smr.1939 · Cited by 20
A Review of the Applications of Affordance Theory in mHealth App Research
Alshawmar, M. (2021) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2021.437 · Cited by 22
Saidani, I., Ouni, A., Mkaouer, M. W., & Palomba, F. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106618 · Cited by 20
Log-based slicing for system-level test cases
Messaoudi, S., Shin, D., Panichella, A., Bianculli, D., & Briand, L. (2021) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3460319.3464824 · Cited by 20
Managing Quality Assurance Challenges of DevOps through Analytics
Ibrahim, M. M. A., Syed-Mohamad, S. M., & Husin, M. H. (2019) · International Conference on Software and Computer Applications (ICSCA) · doi:10.1145/3316615.3316670 · Cited by 19
Utilising CI environment for efficient and effective testing of NFRs
Yu, L., Alégroth, E., Chatzipetrou, P., & Gorschek, T. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2019.106199 · Cited by 21
Continuous Delivery of Blockchain Distributed Applications
Górski, T. (2021) · Sensors · doi:10.3390/s22010128 · Cited by 19
Applying Test Driven Development in the Big Data Domain – Lessons From the Literature
Staegemann, D., Volk, M., Lautenschlager, E., Pohl, M., Abdallah, M., & Turowski, K. (2021) · 2021 International Conference on Information Technology (ICIT) · doi:10.1109/icit52682.2021.9491728 · Cited by 19
L’Erário, A., Thomazinho, H. C. S., & Fabri, J. A. (2020) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194020500217 · Cited by 18
TCTracer: Establishing test-to-code traceability links using dynamic and static techniques
White, R., & Krinke, J. (2022) · Empirical Software Engineering · doi:10.1007/s10664-021-10079-1 · Cited by 18
Metadata
- Authors
- Shahin, M., Ali Babar, M., & Zhu, L.
- Affiliations
- The University of Adelaide (AU)
- Commonwealth Scientific and Industrial Research Organisation (AU)
- Venue
- IEEE Access
- Published
- 2017
- Volume
- 5
- Pages
- 3909-3943
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W2603119212
- Cited by
- 680
- References
- 47
- Citations by year
- 2026: 86 · 2025: 111 · 2024: 89 · 2023: 97 · 2022: 89 · 2021: 83 · 2020: 49 · 2019: 38 · 2018: 33 · 2017: 5
- FWCI
- 10.80
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion)
- arXiv (Cornell University) (acceptedVersion)
- RMIT Research Repository (RMIT University Library) (acceptedVersion)
- Adelaide Research & Scholarship (AR&S) (University of Adelaide) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- arXiv (Cornell University)
- Funders
- Data61, a business unit of CSIRO, Australia
- Australian Government Research Training Program Scholarship
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- continuous integration, continuous delivery, continuous deployment, continuous practice, deployment pipelines, scalability, barriers to adoption
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
