CoChat
Sign Up
Cited by 484Software Engineering Techniques and PracticesSoftware System Performance and ReliabilitySoftware Engineering Research

A Survey of DevOps Concepts and Challenges

Open access
Leite, Leonardo, Rocha, Carla, Kon, Fabio, Milojicic, Dejan, Meirelles, Paulo·2020·ACM Computing Surveys
DOI 10.1145/3359981
Open this paper in CoChat — free accountOpen PDF

Abstract

DevOpsis a collaborative and multidisciplinary organizational effort to automate continuous delivery of new software updates while guaranteeing their correctness and reliability. The present survey investigates and discusses DevOps challenges from the perspective of engineers, managers, and researchers. We review the literature and develop a DevOps conceptual map, correlating the DevOps automation tools with these concepts. We then discuss their practical implications for engineers, managers, and researchers. Finally, we critically explore some of the most relevant DevOps challenges reported by the literature.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3359981

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • There is strong agreement that continuous security validation and integration are essential to close potential security gaps as code progresses through the development pipeline.

    “From Leite et al. [21] through Nisha & Khandebharad [31], there is strong agreement that continuous security validation and integration are essential to close potential security gaps as code progresses through the development pipeline.”

    From Evolution of DevSecOps and Its Influence on Application Security: A Systematic Literature Review (2025) · cites this paper as [21]

    Section 6.3 (p. 18) says source-code analysis tools can identify vulnerabilities, and Section 6.4 (p. 18) describes automated tests in continuous-integration pipelines. Section 7.1 (p. 23) also calls for security to be addressed in software implementation. The paper does not establish strong agreement that continuous security validation throughout the pipeline is essential to close security gaps. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • The papers stress that security controls should be integrated at every stage of development.

    “Particularly in the works of Leite et al. [21], Rajapakse et al. [28], and Akbar et al. [29], the papers stress that security controls should be integrated at every stage of development.”

    From Evolution of DevSecOps and Its Influence on Application Security: A Systematic Literature Review (2025) · cites this paper as [21]

    Section 7.1 (p. 23) says developers must be educated in software security and that security requirements must be addressed in software implementation, including deployment-automation code. It does not say security controls should be integrated at every stage of development. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 35

  1. [51]2017. xMatters Atlassian DevOps Maturity Survey Report 2017. (2017). https://www.xmatters.com/press-release/ xmatters-atlassian-2017-devops-maturity-survey-report/, accessed on Jun 2018.
  2. [55]Len Bass, Ingo Weber, and Liming Zhu. 2015. DevOps: A Software Architect's Perspective . Addison-Wesley Professional.
  3. [57]Kent Beck and Cynthia Andres. 2004. Extreme programming explained: embrace change . Addison-Wesley Professional.
  4. [58]Betsy Beyer, Chris Jones, Jennifer Petoff, and Niall Richard Murphy. 2016. Site Reliability Engineering: How Google Runs Production Systems . O'Reilly Media.
  5. [62]David Budgen and Pearl Brereton. 2006. Performing Systematic Literature Reviews in Software Engineering. In Proceedings of the 28th International Conference on Software Engineering (ICSE '06) . ACM, 1051-1052.
  6. [64]Kathy Charmaz. 2008. Chapter 7: Grounded Theory as an Emergent Method. In Handbook of Emergent Methods . The Guilford Press.
  7. [66]Juliet Corbin and Anselm Strauss. 2014. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory (4th ed.). SAGE Publications, Inc.
  8. [67]Breno B. Nicolau de França, Helvio Jeronimo, Junior, and Guilherme Horta Travassos. 2016. Characterizing DevOps by Hearing Multiple Voices. In Proceedings of the 30th Brazilian Symposium on Software Engineering (SBES '16) . ACM, 53-62.
  9. [70]Floris Erich, Chintan Amrit, and Maya Daneva. 2014. A Mapping Study on Cooperation between Information System Development and Operations. In Product-Focused Software Process Improvement , Andreas Jedlitschka, Pasi Kuvaja, Marco Kuhrmann, Tomi Männistö, Jürgen Münch, and Mikko Raatikainen (Eds.). Springer International Publishing, Cham, 277-280.
  10. [71]F. M. A. Erich, C. Amrit, and M. Daneva. 2017. A Qualitative Study of DevOps Usage in Practice. Journal of Software: Evolution and Process 29, 6 (2017), e1885.
  11. [72]Nicole Forsgren, Jez Humble, and Gene Kim. 2018. Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations . IT Revolution Press.
  12. [76]Peter J Hager, Howard Jeffrey Scheiber, and Nancy C Corbin. 1997. Designing & delivering: Scientific, technical, and managerial presentations . John Wiley & Sons.
  13. [75]Georges Bou Ghantous and Asif Gill. 2017. DevOps: Concepts, Practices, Tools, Benefits and Challenges. In 21st Pacific Asia Conference on Information Systems (PACIS 2017) . 96:1-96:12.
  14. [77]James Hamilton. 2007. On Designing and Deploying Internet-Scale Services. In Proceedings of the 21st Large Installation System Administration Conference (LISA '07) . USENIX, 231-242.
  15. [82]Jez Humble and David Farley. 2010. Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation . Addison-Wesley Professional.
  16. [83]Ramtin Jabbari, Nauman bin Ali, Kai Petersen, and Binish Tanveer. 2016. What is DevOps?: A Systematic Mapping Study on Definitions and Practices. In Proceedings of the Scientific Workshop Proceedings of XP2016 (XP '16 Workshops) . ACM, 12:1-12:11.
  17. [86]N. Kerzazi and B. Adams. 2016. Botched Releases: Do We Need to Roll Back? Empirical Study on a Commercial Web App. In 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER) , Vol. 1. 574-583.
  18. [87]N. Kerzazi and B. Adams. 2016. Who Needs Release and DevOps Engineers, and Why?. In 2016 IEEE/ACM International Workshop on Continuous Software Evolution and Delivery (CSED) . 77-83.
  19. [90]Gene Kim, Jez Humble, Patrick Debois, and John Willis. 2016. The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations . IT Revolution Press.
  20. [92]Per Kroll and Philippe Kruchten. 2003. The rational unified process made easy: a practitioner's guide to the RUP . Addison-Wesley Professional.
  21. [96]Robert C. Martin. 2008. Chapter 12: Emergence. In Clean Code: A Handbook of Agile Software Craftsmanship . Prentice Hall.
  22. [95]Lucy Ellen Lwakatare, Pasi Kuvaja, and Markku Oivo. 2015. Dimensions of DevOps. In Agile Processes in Software Engineering and Extreme Programming . Springer International Publishing, 212-217.
  23. [97]M. Douglas McIlroy, J. M. Buxton, Peter Naur, and Brian Randell. 1968. Mass-produced software components. In Software Engineering Concepts and Techniques, 1968 NATO Conference on Software Engineering . 88-98.
  24. [99]Matthew B. Miles and A. Michael Huberman. 1994. Chapter 2: Focusing and Bounding the Collection of Data - the Substantive Start. In Qualitative Data Analysis: An Expanded Sourcebook (6th ed.). SAGE Publications.
  25. [100]Dejan Milojicic. 2011. Autograding in the Cloud: Interview with David O'Hallaron. IEEE Internet Computing 15, 1 (2011), 9-12.
  26. [102]Eueung Mulyana, Rifqy Hakimi, and Hendrawan. 2018. Bringing Automation to the Classroom: A ChatOps-Based Approach. In 2018 4th International Conference on Wireless and Telematics (ICWT) . 1-6.
  27. [101]Kief Morris. 2016. Infrastructure as Code: Managing Servers in the Cloud . O'Reilly Media.
  28. [105]Roger S Pressman. 2005. Software engineering: a practitioner's approach (6th ed.). Palgrave Macmillan.
  29. [104]Mary Poppendieck and Tom Poppendieck. 2006. Implementing Lean Software Development: From Concept to Cash . Addison-Wesley Professional.
  30. [111]Alexandra Sbaraini, Stacy M Carter, R Wendell Evans, and Anthony Blinkhorn. 2011. How to do a grounded theory study: a worked example of a study of dental practices. BMC medical research methodology 11, 128 (2011), 1-20.
  31. [110]Danilo Sato. 2014. Chapter 12: Infrastructure as Code. In Devops in Practice: Reliable and automated software delivery . Casa do Código.
  32. [114]Jens Smeds, Kristian Nybom, and Ivan Porres. 2015. DevOps: A Definition and Perceived Adoption Impediments. In Agile Processes in Software Engineering and Extreme Programming . Springer International Publishing, 166-177.
  33. [115]Ian Sommerville. 2011. Software engineering (9th ed.). Addison-Wesley.
  34. [116]Daniel Stahl, Torvald Martensson, and Jan Bosch. 2017. Continuous practices and devops: beyond the buzz, what does it all mean?. In 43rd Euromicro Conference on Software Engineering and Advanced Applications (SEAA 2017) . 440-448.
  35. [118]Antonio Terceiro, Joenio Costa, João Miranda, Paulo Meirelles, Luiz Romário Rios, Lucianna Almeida, Christina Chavez, and Fabio Kon. 2010. Analizo: an extensible multi-language source code analysis and visualization toolkit. In Brazilian Conference on Software: Theory and Practice (Tools Session) (CBSoft) , Vol. 29.

Cited by

Sorted by citations

Authors
Leite, L., Rocha, C., Kon, F., Milojicic, D., & Meirelles, P.
Affiliations
  • Universidade de São Paulo (BR)
  • Universidade de Brasília (BR)
  • University of São, São Paulo, Brazil
  • Hewlett-Packard (United States) (US)
  • Universidade Federal de São Paulo (BR)
ORCID
Venue
ACM Computing Surveys
Publisher
Association for Computing Machinery (ACM)
Published
2019-11-14
Volume
52
Issue
6
Pages
1-35
ISSN
  • 0360-0300 (ISSN-L)
  • 1557-7341
DOI
10.1145/3359981
OpenAlex
W2986642681
Cited by
484
References
3
Citations by year
2026: 41 · 2025: 68 · 2024: 81 · 2023: 105 · 2022: 93 · 2021: 63 · 2020: 32 · 2019: 1
FWCI
58.84
Type
journal article
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • ACM Computing Surveys (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (Finance Code 001)
  • Fundação de Amparo à Pesquisa do Estado de São Paulo (14/50937-1 and 15/24485-9)
  • Conselho Nacional de Desenvolvimento Científico e Tecnológico (465446/2014-0)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
Keywords
DevOps, continuous delivery, software automation, DevOps tools, software reliability, software engineering management

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review