Capabilities and metrics in DevOps: A design science study
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A newly published paper by Amaro et al. (2023) elicited 24 DevOps metrics through MLR and interview.
“Moreover, a newly published paper by Amaro et al. (2023) elicited 24 DevOps metrics through MLR and interview.”
From Identifying the primary dimensions of DevSecOps: A multi-vocal literature review (2024) · cites this paper as Amaro et al., 2023
Cites
Saturation in qualitative research: exploring its conceptualization and operationalization
Saunders, B., Sim, J., Kingstone, T., Baker, S., Waterfield, J., Bartlam, B., Burroughs, H., & Jinks, C. (2018) · Quality & Quantity · doi:10.1007/s11135-017-0574-8 · Cited by 12,033 · see citing works
Design Science in Information Systems Research1
Hevner, A. R., March, S. T., Park, J., & Ram, S. (2004) · MIS Quarterly · doi:10.2307/25148625 · Cited by 10,707 · see citing works
(2004) · Cited by 5,198 · see citing works
Design and natural science research on information technology
March, S. T., & Smith, G. F. (1995) · Decision Support Systems · doi:10.1016/0167-9236(94)00041-2 · Cited by 3,871 · see citing works
A simple method to assess and report thematic saturation in qualitative research
Guest, G., Namey, E., & Chen, M. (2020) · PLOS ONE · doi:10.1371/journal.pone.0232076 · Cited by 2,929 · see citing works
Contributing Knowledge to Electronic Knowledge Repositories: An Empirical Investigation1
Kankanhalli, A., Tan, B. C. Y., & Wei, K. (2005) · MIS Quarterly · doi:10.2307/25148670 · Cited by 2,831 · see citing works
Conducting Semi‐Structured Interviews
Adams, W. C. (2015) · Wiley eBooks · doi:10.1002/9781119171386.ch19 · Cited by 1,692 · see citing works
A Three Cycle View of Design Science Research
Hevner, A. R. (2007) · Journal of the Association for Information Systems · Cited by 1,645 · see citing works
(2007) · Journal of the Association for Information Systems · Cited by 1,638 · see citing works
Does Sample Size Matter in Qualitative Research?: A Review of Qualitative Interviews in is Research
Marshall, B., Cardon, P., Poddar, A., & Fontenot, R. (2013) · Journal of Computer Information Systems · doi:10.1080/08874417.2013.11645667 · Cited by 1,636 · see citing works
FEDS: a Framework for Evaluation in Design Science Research
Venable, J. R., Pries‐Heje, J., & Baskerville, R. (2014) · European Journal of Information Systems · doi:10.1057/ejis.2014.36 · Cited by 1,063 · see citing works
Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation
Humble, J., & Farley, D. R. (2010) · Cited by 911 · see citing works
Situating and Constructing Diversity in Semi-Structured Interviews
McIntosh, M., & Morse, J. M. (2015) · Global Qualitative Nursing Research · doi:10.1177/2333393615597674 · Cited by 873 · see citing works
Garousi, V., Felderer, M., & Mäntylä, M. V. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2018.09.006 · Cited by 831 · see citing works
A Comprehensive Framework for Evaluation in Design Science Research
Venable, J. R., Pries‐Heje, J., & Baskerville, R. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-29863-9_31 · Cited by 462 · see citing works
Toyota Kata: Managing People for Improvement, Adaptiveness and Superior Results
Rother, M. (2009) · Cited by 421 · see citing works
Strategies for Design Science Research Evaluation
Pries‐Heje, J., Baskerville, R., & Venable, J. R. (2008) · Journal of the Association for Information Systems · Cited by 328 · see citing works
20th Americas Conference on Information Systems, AMCIS 2014
Rahimi, F., Møller, C., & Hvam, L. (2014) · Americas Conference on Information Systems · Cited by 259 · see citing works
Towards Rigor in Reviews of Multivocal Literatures: Applying the Exploratory Case Study Method
Ogawa, R. T., & Malen, B. (1991) · Review of Educational Research · doi:10.3102/00346543061003265 · Cited by 247 · see citing works
A typology of organisational cultures
Westrum, R. (2004) · BMJ Quality & Safety · doi:10.1136/qshc.2003.009522 · Cited by 247 · see citing works
Pries-Heje, & Baskerville, R. (2008) · MIS Quarterly · doi:10.2307/25148870 · Cited by 241 · see citing works
(1998) · Cited by 216 · see citing works
Garousi, V., Felderer, M., & Mäntylä, M. V. (2016) · Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2915970.2916008 · Cited by 210 · see citing works
A qualitative study of DevOps usage in practice
Erich, F., Amrit, C., & Daneva, M. (2017) · Journal of Software Evolution and Process · doi:10.1002/smr.1885 · Cited by 208 · see citing works
Site Reliability Engineering: How Google Runs Production Systems
Beyer, B., Jones, C., Petoff, J. L. M., & Murphy, N. R. (2016) · Cited by 206 · see citing works
Kim, G. H., Debois, P., Willis, J., & Humble, J. (2016) · Cited by 205 · see citing works
The 17th European conference on information systems (ECIS)
Paz, A. I. L., & Seo, D. (2009) · European Conference on Information Systems · Cited by 200 · see citing works
DevOps and software quality: A systematic mapping
Mishra, A., & Otaiwi, Z. (2020) · Computer Science Review · doi:10.1016/j.cosrev.2020.100308 · Cited by 177 · see citing works
DevOps Capabilities, Practices, and Challenges: Insights from a Case Study
Senapathi, M., Buchan, J., & Osman, H. (2018) · Proceedings of the 22nd International Conference on Evaluation and Assessment in Software Engineering 2018 · doi:10.1145/3210459.3210465 · Cited by 172 · see citing works
Genres of Inquiry in Design-Science Research: Justification and Evaluation of Knowledge Production1
Baskerville, R., Kaul, M., & Storey, V. C. (2015) · MIS Quarterly · doi:10.25300/misq/2015/39.3.02 · Cited by 164 · see citing works
DevOps: A Definition and Perceived Adoption Impediments
Smeds, J., Nybom, K., & Porres, I. (2015) · Lecture Notes in Business Information Processing · doi:10.1007/978-3-319-18612-2_14 · Cited by 158 · see citing works
Evaluation Patterns for Design Science Research Artefacts
Sonnenberg, C., & Brocke, J. V. (2012) · Communications in computer and information science · doi:10.1007/978-3-642-33681-2_7 · Cited by 156 · see citing works
Towards Definitions for Release Engineering and DevOps
Dyck, A., Penners, R., & Lichter, H. (2015) · doi:10.1109/releng.2015.10 · Cited by 154 · see citing works
Conducting Semi-structured Interviews
Raworth, K., Narayan, S., Sweetman, C., Rowlands, J., & Hopkins, A. (2012) · Cited by 130 · see citing works
Characterizing DevOps by Hearing Multiple Voices
de França, B. B. N., Jeronimo, H., & Travassos, G. H. (2016) · doi:10.1145/2973839.2973845 · Cited by 109 · see citing works
2016 2nd International Conference on Open and Big Data (OBD)
(2016) · doi:10.1109/obd38069.2016 · Cited by 84 · see citing works
Advances in Using Agile and Lean Processes for Software Development
Rodríguez, P., Mäntylä, M., Oivo, M., Lwakatare, L. E., Seppänen, P., & Kuvaja, P. (2018) · Advances in computers · doi:10.1016/bs.adcom.2018.03.014 · Cited by 70 · see citing works
Self-Service Cybersecurity Monitoring as Enabler for DevSecOps
Díaz, J., Pérez, J., Peña, M. Á. L., Mena, G. A., & Yagüe, A. (2019) · IEEE Access · doi:10.1109/access.2019.2930000 · Cited by 64 · see citing works
Tan, F. T. C., Pan, S. L., & Zuo, M. (2018) · Information Systems Journal · doi:10.1111/isj.12221 · Cited by 55 · see citing works
Characteristics of Open Data CSV Files
Mitlöhner, J., Neumaier, S., Jürgen Umbrich, A., & Polleres, A. (2016) · International Conference on Open and Big Data (OBD) · doi:10.1109/obd.2016.18 · Cited by 53 · see citing works
Preliminary Findings about DevSecOps from Grey Literature
Mao, R., Zhang, H., Dai, Q., Huang, H., Rong, G., Shen, H., Chen, L., & Lu, K. (2020) · 2020 IEEE 20th International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs51102.2020.00064 · Cited by 49 · see citing works
DevOpSlang – Bridging the Gap between Development and Operations
Wettinger, J., Breitenbücher, U., & Leymann, F. (2014) · Lecture notes in computer science · doi:10.1007/978-3-662-44879-3_8 · Cited by 47 · see citing works
Pereira, R., & Serrano, J. P. (2020) · Journal of Information Technology · doi:10.1177/0268396219886874 · Cited by 46 · see citing works
Prates, L., Faustino, J., Silva, M., & Pereira, R. (2019) · Lecture Notes in Business Information Processing · doi:10.1007/978-3-030-29608-7_7 · Cited by 45 · see citing works
Sun, D. W., Fu, M., Zhu, L., Li, G., & Lu, Q. (2016) · IEEE Transactions on Emerging Topics in Computing · doi:10.1109/tetc.2016.2520883 · Cited by 44 · see citing works
The Oxford Handbook of Qualitative Research in American Music Education
(2014) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199844272.001.0001 · Cited by 39 · see citing works
Lean Enterprise: How High Performance Organizations Innovate at Scale
Humble, J., Molesky, J., & O'Reilly, B. (2015) · Cited by 37 · see citing works
Using Analytics to Guide Improvement during an Agile–DevOps Transformation
Snyder, B., & Curtis, B. (2017) · IEEE Software · doi:10.1109/ms.2017.4541032 · Cited by 37 · see citing works
Mäkinen, S., Leppänen, M., Kilamo, T., Mattila, A.-L., Laukkanen, E., Pagels, M., & Männistö, T. (2016) · Information and Software Technology · doi:10.1016/j.infsof.2016.09.001 · Cited by 34 · see citing works
DevOpRET: Continuous reliability testing in DevOps
Bertolino, A., Angelis, G. D., Guerriero, A., Miranda, B., Pietrantuono, R., & Russo, S. (2020) · Journal of Software Evolution and Process · doi:10.1002/smr.2298 · Cited by 32 · see citing works
Practical Aspects of Design Science
Helfert, M., & Donnellan, B. (2012) · Communications in computer and information science · doi:10.1007/978-3-642-33681-2 · Cited by 30 · see citing works
Teixeira, D., Pereira, R., Henriques, T. A., da Silva, M. M., Jo�ã, N. A., Faustino, J., & da Silva, M. M. (2020) · International Journal of Agile Systems and Management · doi:10.1504/ijasm.2020.112343 · Cited by 27 · see citing works
A DevOps Implementation Framework for Large Agile-Based Financial Organizations
Nagarajan, A. D., & Overbeek, S. J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-02610-3_10 · Cited by 27 · see citing works
An Agile Framework for ITS Management In Organizations
Sahid, A., Maleh, Y., & Belaissaoui, M. (2017) · doi:10.1145/3167486.3167556 · Cited by 26 · see citing works
38. Management Information Systems: Managing the Digital Firm : Global Edition.
Laudon, K. C., & Laudon, J. P. (2019) · Cited by 24 · see citing works
The Role of Continuous Delivery in IT and Organizational PerformancePreprint
Forsgren, N., & Humble, J. (2015) · SSRN Electronic Journal · doi:10.2139/ssrn.2681909 · Cited by 24 · see citing works
Towards a DevOps Approach for Software Quality Engineering
Perez, J. F., Wang, W., & Casale, G. (2015) · doi:10.1145/2693561.2693564 · Cited by 23 · see citing works
The DevOps Adoption Playbook: A Guide to Adopting DevOps in a Multi-Speed IT Enterprise
Sharma, S. (2017) · Cited by 22 · see citing works
A Systematic Literature Review on DevOps Capabilities and Areas
Teixeira, D., Pereira, R., Henriques, T. A., da Silva, M. M., & Faustino, J. (2020) · International Journal of Human Capital and Information Technology Professionals · doi:10.4018/ijhcitp.2020040101 · Cited by 22 · see citing works
DevOps with Continuous Testing Architecture and Its Metrics Model
Angara, J., Gutta, S., & Prasad, S. (2018) · Advances in intelligent systems and computing · doi:10.1007/978-981-10-8633-5_28 · Cited by 20 · see citing works
Managing Quality Assurance Challenges of DevOps through Analytics
Ibrahim, M. M. A., Syed-Mohamad, S. M., & Husin, M. H. (2019) · International Conference on Software and Computer Applications (ICSCA) · doi:10.1145/3316615.3316670 · Cited by 19 · see citing works
A Multivocal Literature Review on the use of DevOps for e-Learning systems
Sánchez‐Gordón, M., & Colomo‐Palacios, R. (2018) · Ninth International Conference on Technological Ecosystems for Enhancing Multiculturality (TEEM'21) · doi:10.1145/3284179.3284328 · Cited by 16 · see citing works
Software Test Automation Maturity: A Survey of the State of the Practice
Wang, Y., Mäntylä, M., Demeyer, S., Wiklund, K., Eldh, S., & Kairi, T. (2020) · International Conference on Software Technologies (ICSOFT) · doi:10.5220/0009766800270038 · Cited by 16 · see citing works
Conducting and Analyzing Individual Interviews
Roulston, K. (2014) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199844272.013.014 · Cited by 14 · see citing works
Specifying the Process Model for Systematic Reviews: An Augmented Proposal
Olsina, L. A., Becker, P., Peppino, D., & Tebes, G. (2019) · Journal of Software Engineering Research and Development · doi:10.5753/jserd.2019.460 · Cited by 11 · see citing works
Ding, Z. (2019) · Spectrum Research Repository (Concordia University) · Cited by 11 · see citing works
Misunderstood as mothers: women's stories of being hospitalized for illness in the postpartum period
Power, T., Jackson, D. W., Carter, B., & Weaver, R. (2014) · Journal of Advanced Nursing · doi:10.1111/jan.12515 · Cited by 8 · see citing works
A Taxonomy of Software Delivery Performance Profiles: Investigating the Effects of DevOps Practices
Forsgren, N., Rothenberger, M. A., Humble, J., Thatcher, J. B., & Smith, D. (2020) · Journal of the Association for Information Systems · Cited by 7 · see citing works
Tamburri, D. A., Nucci, D. D., Giacomo, L. D., & Palomba, F. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-06019-0_4 · Cited by 6 · see citing works
Dogfooding: using IBM cloud services to monitor IBM cloud infrastructure.Preprint
Pourmajidi, W., Miranskyy, A. V., Steinbacher, J., Erwin, T., & Godwin, D. (2019) · arXiv (Cornell University) · Cited by 6 · see citing works
DevOps Enhancement with Continuous Test Optimization
Marijan, D., & Sen, S. (2018) · Proceedings/Proceedings of the ... International Conference on Software Engineering and Knowledge Engineering · doi:10.18293/seke2018-168 · Cited by 5 · see citing works
DevOps Capability Assessment in a Software Development Team
Kurkela, M. (2020) · Theseus (Ammattikorkeakoulujen) · Cited by 3 · see citing works
Cuppett, M. S. (2016) · Apress eBooks · doi:10.1007/978-1-4842-2208-9_2 · Cited by 2 · see citing works
Kuiper, C. J. (2019) · Digital Commons (Liberty University) · Cited by 1 · see citing works
Flagged first, then by citations
3 of the 78 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Identifying the primary dimensions of DevSecOps: A multi-vocal literature review
Zhao, X., Clear, T., & Lal, R. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112063 · Cited by 40 · see citing works
Theory and practice in secure software development lifecycle: A comprehensive survey
Otieno, M., Odera, D., & Ounza, J. E. (2023) · World Journal of Advanced Research and Reviews · doi:10.30574/wjarr.2023.18.3.0944 · Cited by 29 · see citing works
Mapping DevOps capabilities to the software life cycle: A systematic literature review
Amaro, R., Pereira, R., & da Silva, M. M. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107583 · Cited by 28 · see citing works
Sanmocte, E. M. T., & Costales, J. A. (2025) · International Journal of Computer Theory and Engineering · doi:10.7763/ijcte.2025.v17.1367 · Cited by 7 · see citing works
Enhancing Expert Interviews: Insights from IS and Digital Transformation Research
Hassmann, T., & Westner, M. K. (2024) · Proceedings of the International Conference on Information Systems Development · doi:10.62036/isd.2024.48 · Cited by 6 · see citing works
Heuristic evaluation of the sport analysis application interface
Ihsan, N., Okilanda, A., Sepriadi, S., Farell, G., Shapie, M. N. M., & Zakaria, J. (2024) · Retos · doi:10.47197/retos.v54.103272 · Cited by 5 · see citing works
Refining the Scrum Paradigm: A Comprehensive Research of Software Development Practices (2020–2023)
Pashchenko, D. S. (2024) · Computing&AI Connect · doi:10.69709/caic.2024.103102 · Cited by 5 · see citing works
DevOps research across a decade: Patterns, trends, and insights with LDA
Patel, A. R., Tyagi, S., & Sharma, D. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108118 · Cited by 2 · see citing works
Assessing the Sustainable Transition in LSPs Industry
Centobelli, P. (2025) · Corporate Social Responsibility and Environmental Management · doi:10.1002/csr.3161 · Cited by 2 · see citing works
Toward the Implementation of DevOps: A Guide of Tools, Practices and Activities
Acevedo, D., Muñoz, M., Galvan, S., Mejía, J., & Robles, B. D. V. (2025) · Communications in computer and information science · doi:10.1007/978-3-032-04288-0_7 · Cited by 1 · see citing works
An Agile and Adaptable DevOps Framework With Tunable Metrics for Enhanced Process Efficiency
Altuwaijri, F. S. (2026) · IEEE Access · doi:10.1109/access.2026.3651953 · Cited by 1 · see citing works
A Practical Approach of Devsecops Integration in Multi-Tier Web ApplicationsPreprint
Rida, A., Bakhil, A., & Lahcen, A. A. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5337693
Maintainability Measurement Framework in DevOps Settings: A Proposal
García‐Mireles, G. A., Olivero, N. P., & Ávila-George, H. (2025) · Studies in computational intelligence · doi:10.1007/978-3-031-90310-6_5
EXPLORING THE IMPACT OF DEVOPS MATURITY ON PROJECT SUCCESS: A SYSTEMATIC LITERATURE REVIEW
Madhushan, T. (2025) · Desk Research Conference · doi:10.64920/drc2025073
Kumar, A. (2025) · International Journal of Scientific Research and Modern Technology. · doi:10.38124/ijsrmt.v4i10.1434
CONCEPTUAL MODEL FOR THE USE OF DEVOPS AND AI TOOLS IN AGILE PROJECTSPreprint
Feitosa, D. N. M., Penha, R., Filho, J. R. D. C., Galvão, G. S., & da Silva, V. F. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6956551
Yeşilyayla, H. (2026) · Journal of Business Innovation and Governance · doi:10.54472/jobig.1921341
Empirical Evaluation of a DevSecOps Proxy Pipeline for Multi-Tier Web Applications
Rida, A., Bakhil, A., & Lahcen, A. A. (2026) · Future Internet · doi:10.3390/fi18070371
DevSecOps: A Comprehensive Survey of Secure Software Development and Deployment Practices
Shrivastava, D. P. K. (2026) · International Journal of Advanced Research in Science Communication and Technology · doi:10.48175/ijarsct-38515
Ullah, S. (2026) · Journal of Social Research Development · doi:10.59075/jsrd.v7i1.591
Syahputra, M. F., Sarno, R., Febryanto, K. C., Sungkono, K. R., Fauzi, A. A., & Setiawan, H. (2026) · Electrical Power, Electronics, Communications, Controls and Informatics Seminar (EECCIS) · doi:10.1109/eeccis68982.2026.11750573
- Authors
- Amaro, R., Pereira, R., & da Silva, M. M.
- Affiliations
- Iscte – Instituto Universitário de Lisboa (PT)
- Corresponding author
- Amaro, Ricardo
- Venue
- Information & Management
- Publisher
- Elsevier BV
- Published
- 2023-07
- Volume
- 60
- Issue
- 5
- Pages
- 103809
- ISSN
- 0378-7206 (ISSN-L)
- OpenAlex
- W4375851246
- Cited by
- 21
- References
- 78
- Citations by year
- 2026: 8 · 2025: 7 · 2024: 5 · 2023: 1
- FWCI
- 3.80
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- Information & Management (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Business, Management and Accounting › Management Information Systems › Information Technology Governance and Strategy
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Keywords
- DevOps, DevOps adoption, IT organizations, organizational culture, team empowerment, capability measurement
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
