CoChat

Measuring Software Delivery Performance Using the Four Key Metrics of DevOps

Open access
Sallin, Marc, Kropp, Martin, Anslow, Craig, Quilty, James W., Meier, Andreas · 2021 · Lecture Notes in Business Information Processing

The DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index; 2 of the 12 sources its claims cite were checked for support.

Abstract

The Four Key Metrics of DevOps have become very popular for measuring IT-performance and DevOps adoption.

Overview

CoChat Summary

TL;DR

The paper investigates how the four key DevOps metrics can be measured automatically and argues that the speed metrics are easier to automate than the stability metrics. In a Swiss Post case study, a prototype and survey suggested the automatic measurement was useful and that participants found the metrics valid, though incomplete.

Key findings

  • The multivocal literature review found no scientific literature on automatic measurement of the four key metrics, but 16 unique gray-literature articles relevant to definitions or experience reports.
  • In the gray literature, 7/16 articles defined deployment frequency, 9/16 lead time for change, 8/16 time to restore service, and 9/16 change failure rate.
  • The prototype measured the metrics using the team’s toolchain: Jenkins, PowerShell, Splunk, OpenShift, Artifactory, and Jira.
  • In the survey, participants generally agreed the metrics were valid for software delivery performance, and the recommendation score for using the automatic measurement was 8.3/10 (max 10, min 6, sd 1.5).

Methods

  • The study combined a multivocal literature review and an industrial case study at Swiss Post.
  • For the case study, the prototype was used by a Scrum team of 10 people for three weeks; 6 of 10 team members completed the survey.
  • Analysis was based mainly on Likert-scale survey questions with optional free-text responses, plus implementation of automatic metric collection and calculation.

Limitations

  • The paper states that the SLR part returned no scientific literature, so the review relied on gray literature, which the authors note is less reliable and may omit important sources.
  • The case study had a small, homogeneous sample and was highly context-specific to Swiss Post, limiting generalizability.
  • The team was already a high performer with invested CI/CD processes, and the authors note the measurement period was too short to ask about actual effects; these are limitations evident from the text, in addition to those explicitly stated.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/978-3-030-78098-2_7

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

2 Valid · 2 Not valid · 8 Not checked · 57 Own results and interpretations

10 more claims

Create a free account to see all 69 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 38
  • It may state two separate claims as one. 34

Create a free account to see these claims.

Citations

Cites

Show all 9 references

Research works printed but not linked: 2

  1. 10. Forsgren, N., Humble, J., Kim, G.: Accelerate: The Science of Lean Software and DevOps. 1st edn. IT Revolution Press (2018)
  2. 13. ISO/IEC, IEEE: ISO/IEC/IEEE 24765:2010 - Systems and software engineering Vocabulary. Iso/Iec Ieee 2010 (2010). 410

Cited by

Metadata

Authors
Sallin, M., Kropp, M., Anslow, C., Quilty, J. W., & Meier, A.
Affiliations
  • FHNW University of Applied Sciences and Arts Northwestern Switzerland (CH)
  • Victoria University of Wellington (NZ)
  • ZHAW Zurich University of Applied Sciences (CH)
Venue
Lecture Notes in Business Information Processing
Published
2021
Pages
103-119
ISSN
  • 1865-1348 (ISSN-L)
  • 1865-1356
OpenAlex
W3166620945
Cited by
20
References
8
Citations by year
2026: 4 · 2025: 4 · 2024: 4 · 2023: 4 · 2022: 4
FWCI
2.48
Type
conference paper
Language
en
Open access
Licence
Locations
  • Lecture notes in business information processing (publishedVersion, cc-by)
  • Zürcher Hochschule für Angewandte Wissenschaften digital collection (Zurich University of Applied Sciences) (submittedVersion, cc-by-nc-nd)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
Keywords
deployment frequency, DevOps adoption, IT performance measurement

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.