Measuring Software Delivery Performance Using the Four Key Metrics of DevOps
Open accessThe DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index; 2 of the 12 sources its claims cite were checked for support.
Abstract
The Four Key Metrics of DevOps have become very popular for measuring IT-performance and DevOps adoption.
Overview
CoChat Summary
TL;DR
The paper investigates how the four key DevOps metrics can be measured automatically and argues that the speed metrics are easier to automate than the stability metrics. In a Swiss Post case study, a prototype and survey suggested the automatic measurement was useful and that participants found the metrics valid, though incomplete.
Key findings
- The multivocal literature review found no scientific literature on automatic measurement of the four key metrics, but 16 unique gray-literature articles relevant to definitions or experience reports.
- In the gray literature, 7/16 articles defined deployment frequency, 9/16 lead time for change, 8/16 time to restore service, and 9/16 change failure rate.
- The prototype measured the metrics using the team’s toolchain: Jenkins, PowerShell, Splunk, OpenShift, Artifactory, and Jira.
- In the survey, participants generally agreed the metrics were valid for software delivery performance, and the recommendation score for using the automatic measurement was 8.3/10 (max 10, min 6, sd 1.5).
Methods
- The study combined a multivocal literature review and an industrial case study at Swiss Post.
- For the case study, the prototype was used by a Scrum team of 10 people for three weeks; 6 of 10 team members completed the survey.
- Analysis was based mainly on Likert-scale survey questions with optional free-text responses, plus implementation of automatic metric collection and calculation.
Limitations
- The paper states that the SLR part returned no scientific literature, so the review relied on gray literature, which the authors note is less reliable and may omit important sources.
- The case study had a small, homogeneous sample and was highly context-specific to Swiss Post, limiting generalizability.
- The team was already a high performer with invested CI/CD processes, and the authors note the measurement period was too short to ask about actual effects; these are limitations evident from the text, in addition to those explicitly stated.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 2 Not valid · 8 Not checked · 57 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“However, these data may not be complete with respect to the required DevOps aspects (e.g., cultural measures) [7].”
What the sources say
- Supportedcites [7] Forsgren, N., Kersten, M.: DevOps metrics. Queue 15(6), 19–34 (2017) source
In “START BUILDING A BASELINE NOW” (p. 8), the paper says system-based metrics may miss important cultural measures that surveys capture. It also says cultural measures are perceptual and should be measured with surveys. (LLM, conf=0.94)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With the toolchain used by the development team, the measurement of the speed metrics can be completely automated.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We identified no scientific literature which investigated the automatic measurement of the FKM.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The question 'On a scale from 1-10, how likely are you to recommend the metrics measurement to another team?' was answered with an average score of 8.3 (max: 10, min: 6, sd: 1.5).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The analysis shows that the chosen measurement approach is both suitable and the results valuable for the team with respect to measuring and improving the software delivery performance.”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
10 more claims
Create a free account to see all 69 claims and every source verdict.
- Probably not a substantial claim. 38
- It may state two separate claims as one. 34
Citations
Cites
Garousi, V., Felderer, M., & Mäntylä, M. V. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2018.09.006 · Cited by 831
Garousi, V., Felderer, M., & Mäntylä, M. V. (2017) · arXiv.org · doi:10.48550/arxiv.1707.02553 · Cited by 809 · cited 5×
“Thus, stopping criteria need to be defined [8].”
BackgroundMethodInfluentialDevops: A Software Architect's Perspective
Bass, L., Weber, I. M., & Zhu, L. (2015) · CERN Document Server (European Organization for Nuclear Research) · Cited by 561
Garousi, V., Felderer, M., & Mäntylä, M. V. (2016) · Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2915970.2916008 · Cited by 210
Behutiye, W. N., Rodríguez, P., Oivo, M., & Tosun, A. (2017) · Information and Software Technology · doi:10.1016/j.infsof.2016.10.004 · Cited by 116
Forsgren, N., & Kersten, M. (2018) · Communications of the ACM · doi:10.1145/3159169 · Cited by 54
2019 Accelerate State of DevOps Report
Forsgren, N., Smith, D., Humble, J., et al. (2019) · Cited by 25
DevOps Metrics: Your biggest mistake might be collecting the wrong data.
Forsgren, N., & Kersten, M. (2017) · Queue · doi:10.1145/3178368.3182626 · Cited by 5 · cited 2×
“However, these data may not be complete with respect to the required DevOps aspects (e.g., cultural measures) [7].”
Background'Improving ratings': audit in the British university system
Strathern, M. (1997) · European Review · doi:10.1002/(sici)1234-981x(199707)5:3<305::aid-euro184>3.0.co;2-4 · cited 1×
“[18].”
Research works printed but not linked: 2
- Forsgren, N., Humble, J., Kim, G.: Accelerate: The Science of Lean Software and DevOps. 1st edn. IT Revolution Press (2018)
- ISO/IEC, IEEE: ISO/IEC/IEEE 24765:2010 - Systems and software engineering - Vocabulary. Iso/Iec Ieee 2010 (2010). 410
Cited by
Lombardi, F., & Fanton, A. (2023) · Software Quality Journal · doi:10.1007/s11219-023-09619-3 · Cited by 66
“Some effort has been spent in evaluation software delivery performance by means of the four (Sallin et al., 2021).”
MethodDevOps Metrics and KPIs: A Multivocal Literature Review
Amaro, R., Pereira, R., & da Silva, M. M. (2024) · ACM Computing Surveys · doi:10.1145/3652508 · Cited by 51
A Framework for Automating the Measurement of DevOps Research and Assessment (DORA) Metrics
Wilkes, B., Milani, A. M. P., & Storey, M. (2023) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme58846.2023.00018 · Cited by 33
“Sallin et al. mention in their work that the method used for measuring the DORA metrics introduces different strengths and weaknesses [8].”
BackgroundMethodInfluentialOn the need to monitor continuous integration practices
Santos, J., da Costa, D. A., McIntosh, S., & Kulesza, U. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10682-6 · Cited by 15
“In this section, we situate our study with respect to previous works that assessed CI and its impacts.”
On the Benefits of the Accelerate Metrics: An Industrial Survey at Vendasta
Lomio, F., Codabux, Z., Birtch, D., Hopkins, D., & Taibi, D. (2022) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner53432.2022.00017 · Cited by 10
“[10] analyzed the application of the Accelerate metrics in measuring the software delivery performance by automatically calculating the metrics and showing them in production to a team of practitioners.”
Method
Metadata
- Authors
- Sallin, M., Kropp, M., Anslow, C., Quilty, J. W., & Meier, A.
- Affiliations
- FHNW University of Applied Sciences and Arts Northwestern Switzerland (CH)
- Victoria University of Wellington (NZ)
- ZHAW Zurich University of Applied Sciences (CH)
- Venue
- Lecture Notes in Business Information Processing
- Publisher
- Springer International Publishing
- Published
- 2021
- Pages
- 103-119
- ISSN
- 1865-1348 (ISSN-L)
- 1865-1356
- OpenAlex
- W3166620945
- Semantic Scholar
- 1b1bfcf55c6499b1627512df013a8902632c963b
- Cited by
- 20
- References
- 8
- Citations by year
- 2026: 4 · 2025: 4 · 2024: 4 · 2023: 4 · 2022: 4
- FWCI
- 2.48
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor, 160 d)
- Locations
- Lecture notes in business information processing (publishedVersion, cc-by)
- Zürcher Hochschule für Angewandte Wissenschaften digital collection (Zurich University of Applied Sciences) (submittedVersion, cc-by-nc-nd)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- deployment frequency, DevOps adoption, IT performance measurement
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.