CoChat
Sign Up
Cited by 680Software Engineering Techniques and PracticesSoftware System Performance and ReliabilitySoftware Engineering Research

Continuous Integration, Delivery and Deployment: A Systematic Review on Approaches, Tools, Challenges and Practices

Open access
Shahin, Mojtaba, Ali Babar, Muhammad, Zhu, Liming·2017·IEEE Access
DOI 10.1109/access.2017.2685629
Open this paper in CoChat — free account

Abstract

Continuous practices, i.e., continuous integration, delivery, and deployment, are the software development industry practices that enable organizations to frequently and reliably release new features and products. With the increasing interest in the literature on continuous practices, it is important to systematically review and synthesize the approaches, tools, challenges, and practices reported for adopting and implementing continuous practices. This paper aimed at systematically reviewing the state of the art of continuous practices to classify approaches and tools, identify challenges and practices in this regard, and identify the gaps for future research. We used the systematic literature review method for reviewing the peer-reviewed papers on continuous practices published between 2004 and June 1, 2016. We applied the thematic analysis method for analyzing the data extracted from reviewing 69 papers selected using predefined criteria. We have identified 30 approaches and associated tools, which facilitate the implementation of continuous practices in the following ways: (1) reducing build and test time in continuous integration (CI); (2) increasing visibility and awareness on build and test results in CI; (3) supporting (semi-) automated continuous testing; (4) detecting violations, flaws, and faults in CI; (5) addressing security and scalability issues in deployment pipeline; and (6) improving dependability and reliability of deployment process. We have also determined a list of critical factors, such as testing (effort and time), team awareness and transparency, good design principles, customer, highly skilled and motivated team, application domain, and appropriate infrastructure that should be carefully considered when introducing continuous practices in a given organization. The majority of the reviewed papers were validation (34.7%) and evaluation (36.2%) research types. This paper also reveals that continuous practices have been successfully applied to both greenfield and maintenance projects. Continuous practices have become an important area of software engineering research and practice. While the reported approaches, tools, and practices are addressing a wide range of challenges, there are several challenges and gaps, which require future research work for improving the capturing and reporting of contextual information in the studies reporting different aspects of continuous practices; gaining a deep understanding of how software-intensive systems should be (re-) architected to support continuous practices; and addressing the lack of knowledge and tools for engineering processes of designing and running secure deployment pipelines.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/access.2017.2685629

Cites

43 of 43 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Security is among the major concerns that hinder the adoption of DevOps in software enterprises.

    “Despite the popularity of DevOps, security is among the major concerns that hinder the adoption of DevOps in software enterprises [5].”

    From Prioritization based taxonomy of DevOps security challenges using PROMETHEE (2020) · cites this paper as [5]

  • Lack of automated testing tool performance measures for security is a DevOps security challenge.

    “We have also identified lack of automated testing tool performance measures for security [2], [5], [20], [23], [24] as a critical challenge that have a negative impact on DevOps security.”

    From Prioritization based taxonomy of DevOps security challenges using PROMETHEE (2020) · cites this paper as [5]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

4 of the 47 references couldn’t be matched to a known paper.

Research works printed but not linked: 4

  1. [23][23] M. Skelton, and C. O'Dell, Continuous Delivery with Windows and .NET : O'Reilly 2016. -
  2. [28][28] B. A. Kitchenham, T. Dyba, and M. Jorgensen, -EvidenceBased Software Engineering,‖ in 26th International Conference on Software Engineering, 2004, pp. 273-281. -
  3. [29][29] "Cambridge Dictionary, http://dictionary.cambridge.org/." -
  4. [36][36] L. Chen, M. A. Babar, and H. Zhang, -Towards an evidencebased understanding of electronic data sources,‖ in Proceedings of the 14th international conference on Evaluation and Assessment in Software Engineering, UK, 2010, pp. 135-138. -

Cited by

Sorted by citations

Related

Authors
Shahin, M., Ali Babar, M., & Zhu, L.
Affiliations
  • The University of Adelaide (AU)
  • Commonwealth Scientific and Industrial Research Organisation (AU)
ORCID
Venue
IEEE Access
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2017
Volume
5
Pages
3909-3943
ISSN
2169-3536 (ISSN-L)
DOI
10.1109/access.2017.2685629
OpenAlex
W2603119212
Cited by
680
References
47
Citations by year
2026: 86 · 2025: 111 · 2024: 89 · 2023: 97 · 2022: 89 · 2021: 83 · 2020: 49 · 2019: 38 · 2018: 33 · 2017: 5
FWCI
10.80
Type
journal article
Language
en
Open access
Licence
ieeexplore.ieee.org/Xplorehelp/downloads/license-information/OAPA.html (vor)
Locations
  • IEEE Access (publishedVersion)
  • arXiv (Cornell University) (acceptedVersion)
  • RMIT Research Repository (RMIT University Library) (acceptedVersion)
  • Adelaide Research & Scholarship (AR&S) (University of Adelaide) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Figshare (submittedVersion, cc-by)
  • arXiv (Cornell University)
Funders
  • Data61, a business unit of CSIRO, Australia
  • Australian Government Research Training Program Scholarship
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
Keywords
continuous integration, continuous delivery, continuous deployment, continuous practice, deployment pipelines, scalability, barriers to adoption

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review