CoChat
Sign Up
Cited by 680Software Engineering Techniques and PracticesSoftware System Performance and ReliabilitySoftware Engineering Research

The DOI matches its Crossref record; 43 of the 47 references it cites were found in a registry or an index.

Continuous Integration, Delivery and Deployment: A Systematic Review on Approaches, Tools, Challenges and Practices

Open access
Shahin, Mojtaba, Ali Babar, Muhammad, Zhu, Liming·2017·IEEE Access
DOI 10.1109/access.2017.2685629
Open this paper in CoChat — free account

Abstract

Continuous practices, i.e., continuous integration, delivery, and deployment, are the software development industry practices that enable organizations to frequently and reliably release new features and products. With the increasing interest in the literature on continuous practices, it is important to systematically review and synthesize the approaches, tools, challenges, and practices reported for adopting and implementing continuous practices. This paper aimed at systematically reviewing the state of the art of continuous practices to classify approaches and tools, identify challenges and practices in this regard, and identify the gaps for future research. We used the systematic literature review method for reviewing the peer-reviewed papers on continuous practices published between 2004 and June 1, 2016. We applied the thematic analysis method for analyzing the data extracted from reviewing 69 papers selected using predefined criteria. We have identified 30 approaches and associated tools, which facilitate the implementation of continuous practices in the following ways: (1) reducing build and test time in continuous integration (CI); (2) increasing visibility and awareness on build and test results in CI; (3) supporting (semi-) automated continuous testing; (4) detecting violations, flaws, and faults in CI; (5) addressing security and scalability issues in deployment pipeline; and (6) improving dependability and reliability of deployment process. We have also determined a list of critical factors, such as testing (effort and time), team awareness and transparency, good design principles, customer, highly skilled and motivated team, application domain, and appropriate infrastructure that should be carefully considered when introducing continuous practices in a given organization. The majority of the reviewed papers were validation (34.7%) and evaluation (36.2%) research types. This paper also reveals that continuous practices have been successfully applied to both greenfield and maintenance projects. Continuous practices have become an important area of software engineering research and practice. While the reported approaches, tools, and practices are addressing a wide range of challenges, there are several challenges and gaps, which require future research work for improving the capturing and reporting of contextual information in the studies reporting different aspects of continuous practices; gaining a deep understanding of how software-intensive systems should be (re-) architected to support continuous practices; and addressing the lack of knowledge and tools for engineering processes of designing and running secure deployment pipelines.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/access.2017.2685629

Cites

43 of 43 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

4 of the 47 references couldn’t be matched to a known paper.

Research works printed but not linked: 16

  1. 0, Longman Dictionary of Contemporary English Online
  2. luke, 2016, No One Agrees How to Define CI or CD
  3. 0, Camera Dictionary
  4. phillips, 2015, The IT Manager’s Guide to Continuous Delivery Delivering Business Value in Hours
  5. humble, 2016, Continuous delivery vs continuous deployment
  6. 2016, What is Continuous Deployment
  7. humble, 2016, What is Continuous Delivery?
  8. reed, 2016, The Business Case for Continuous Delivery
  9. skelton, 2016, Continuous Delivery With Windows and NET
  10. humble, 2015, Principle 2 Decouple Deployment and Release
  11. 2016, Five Habits of Highly Successful Continuous Delivery Practitioners
  12. thiele, 2016, Continuous Delivery An Easy Must-Have for Agile Development
  13. 2015, State of DevOps Report
  14. sundman, 2013, Continuous delivery vs continuous deployment
  15. suzie, 2016, The Product Managers Guide to Continuous Delivery and DevOps
  16. sato, 2015, Canary Release

Cited by

Sorted by citations

See all 680

Related

See all

Metadata

Authors
Shahin, M., Ali Babar, M., & Zhu, L.
Affiliations
  • The University of Adelaide (AU)
  • Commonwealth Scientific and Industrial Research Organisation (AU)
ORCID
Venue
IEEE Access
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2017
Volume
5
Pages
3909-3943
ISSN
2169-3536 (ISSN-L)
DOI
10.1109/access.2017.2685629
OpenAlex
W2603119212
Cited by
680
References
47
Citations by year
2026: 86 · 2025: 111 · 2024: 89 · 2023: 97 · 2022: 89 · 2021: 83 · 2020: 49 · 2019: 38 · 2018: 33 · 2017: 5
FWCI
10.80
Type
journal article
Language
en
Open access
Licence
ieeexplore.ieee.org/Xplorehelp/downloads/license-information/OAPA.html (vor)
Locations
  • IEEE Access (publishedVersion)
  • arXiv (Cornell University) (acceptedVersion)
  • RMIT Research Repository (RMIT University Library) (acceptedVersion)
  • Adelaide Research & Scholarship (AR&S) (University of Adelaide) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Figshare (submittedVersion, cc-by)
  • arXiv (Cornell University)
Funders
  • Data61, a business unit of CSIRO, Australia
  • Australian Government Research Training Program Scholarship
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
Keywords
continuous integration, continuous delivery, continuous deployment, continuous practice, deployment pipelines, scalability, barriers to adoption

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review