CoChat
Sign Up
Cited by 40Software Engineering Techniques and PracticesSoftware System Performance and ReliabilitySoftware Engineering Research

The DOI matches its Crossref record.

Benchmarking as Empirical Standard in Software Engineering Research

Open access
Hasselbring, Wilhelm·2021·Evaluation and Assessment in Software Engineering
DOI 10.1145/3463274.3463361
Open this paper in CoChat — free accountOpen PDF

Abstract

In empirical software engineering, benchmarks can be used for comparing different methods, techniques and tools. However, the recent ACM SIGSOFT Empirical Standards for Software Engineering Research do not include an explicit checklist for benchmarking. In this paper, we discuss benchmarks for software performance and scalability evaluation as example research areas in software engineering, relate benchmarks to some other empirical research methods, and discuss the requirements on benchmarks that may constitute the basis for a checklist of a benchmarking standard for empirical software engineering research.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3463274.3463361

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 39

  1. 10.1109/65.844498 · DOI
  2. 10.1007/s00778-010-0208-4 · DOI
  3. David Bermbach , Erik Wittern , and Stefan Tai . 2017. Cloud service benchmarking . Springer . David Bermbach, Erik Wittern, and Stefan Tai. 2017. Cloud service benchmarking. Springer.
  4. 10.1016/j.jss.2006.02.063 · DOI
  5. Jim Gray (Ed.). 1993. The Benchmark Handbook for Database and Transaction Systems (2nd ed.). Morgan Kaufmann. Jim Gray (Ed.). 1993. The Benchmark Handbook for Database and Transaction Systems (2nd ed.). Morgan Kaufmann.
  6. 10.1145/2851553.2858659 · DOI
  7. Wilhelm Hasselbring . 2018 . Software Architecture: Past, Present, Future. In The Essence of Software Engineering, Volker Gruhn and Rüdiger Striemer (Eds.) . Springer International Publishing , Cham , 169–184. https://doi.org/10.1007/978-3-319-73897-0_10 10.1007/978-3-319-73897-0_10 Wilhelm Hasselbring. 2018. Software Architecture: Past, Present, Future. In The Essence of Software Engineering, Volker Gruhn and Rüdiger Striemer (Eds.). Springer International Publishing, Cham, 169–184. https://doi.org/10.1007/978-3-319-73897-0_10 · DOI
  8. 10.1109/ICSAW.2017.11 · DOI
  9. Wilhelm Hasselbring and André van Hoorn . 2020 . Kieker: A monitoring framework for software engineering research. Software Impacts 5 (Aug. 2020). https://doi.org/10.1016/j.simpa.2020.100019 10.1016/j.simpa.2020.100019 Wilhelm Hasselbring and André van Hoorn. 2020. Kieker: A monitoring framework for software engineering research. Software Impacts 5 (Aug. 2020). https://doi.org/10.1016/j.simpa.2020.100019 · DOI
  10. 10.1016/j.bdr.2021.100209 · DOI
  11. Nikolas Roman Herbst , Samuel Kounev , and Ralf Reussner . 2013 . Elasticity in Cloud Computing: What It Is, and What It Is Not . In 10th International Conference on Autonomic Computing (ICAC 13) . USENIX Association, San Jose, CA, 23–27. Nikolas Roman Herbst, Samuel Kounev, and Ralf Reussner. 2013. Elasticity in Cloud Computing: What It Is, and What It Is Not. In 10th International Conference on Autonomic Computing (ICAC 13). USENIX Association, San Jose, CA, 23–27.
  12. 10.1109/PDP.2007.70 · DOI
  13. 10.1016/j.sysarc.2008.01.010 · DOI
  14. 10.1007/s10664-016-9483-z · DOI
  15. 10.1109/MCSE.2018.021651343 · DOI
  16. Arne Johanson , Andreas Oschlies , Wilhelm Hasselbring , and Boris Worm . 2017 . SPRAT: A spatially-explicit marine ecosystem model based on population balance equations. Ecological Modelling 349 (April 2017), 11–25. https://doi.org/10.1016/j.ecolmodel.2017.01.020 10.1016/j.ecolmodel.2017.01.020 Arne Johanson, Andreas Oschlies, Wilhelm Hasselbring, and Boris Worm. 2017. SPRAT: A spatially-explicit marine ecosystem model based on population balance equations. Ecological Modelling 349 (April 2017), 11–25. https://doi.org/10.1016/j.ecolmodel.2017.01.020 · DOI
  17. Jeyhun Karimov , Tilmann Rabl , Asterios Katsifodimos , Roman Samarev , Henri Heiskanen , and Volker Markl . 2018 . Benchmarking Distributed Stream Data Processing Systems. In IEEE 34th International Conference on Data Engineering (ICDE). 1507–1518 . https://doi.org/10.1109/ICDE.2018.00169 10.1109/ICDE.2018.00169 Jeyhun Karimov, Tilmann Rabl, Asterios Katsifodimos, Roman Samarev, Henri Heiskanen, and Volker Markl. 2018. Benchmarking Distributed Stream Data Processing Systems. In IEEE 34th International Conference on Data Engineering (ICDE). 1507–1518. https://doi.org/10.1109/ICDE.2018.00169 · DOI
  18. 10.1145/3184407.3184431 · DOI
  19. Holger Knoche and Wilhelm Hasselbring. 2019. Drivers and Barriers for Microservice Adoption - A Survey among Professionals in Germany. Enterprise Modelling and Information Systems Architectures (EMISAJ) - International Journal of Conceptual Modeling 14 1 (2019) 1-35. https://doi.org/10.18417/emisa.14.1 10.18417/emisa.14.1 · DOI
  20. Holger Knoche and Wilhelm Hasselbring. 2019. Drivers and Barriers for Microservice Adoption - A Survey among Professionals in Germany. Enterprise Modelling and Information Systems Architectures (EMISAJ) - International Journal of Conceptual Modeling 14 1 (2019) 1-35. https://doi.org/10.18417/emisa.14.1 · DOI
  21. Samuel Kounev , Klaus-Dieter Lange , and Jóakim von Kistowski . 2020. Systems Benchmarking for Scientists and Engineers . Springer . Samuel Kounev, Klaus-Dieter Lange, and Jóakim von Kistowski. 2020. Systems Benchmarking for Scientists and Engineers. Springer.
  22. 10.1145/3121050.3121099 · DOI
  23. 10.1145/3437479.3437483 · DOI
  24. Paul Ralph Nauman bin Ali Sebastian Baltes Domenico Bianculli Jessica Diaz Yvonne Dittrich Neil Ernst Michael Felderer Robert Feldt Antonio Filieri Breno Bernard Nicolau de França Carlo Alberto Furia Greg Gay Nicolas Gold Daniel Graziotin Pinjia He Rashina Hoda Natalia Juristo Barbara Kitchenham Valentina Lenarduzzi Jorge Martínez Jorge Melegati Daniel Mendez Tim Menzies Jefferson Molleri Dietmar Pfahl Romain Robbes Daniel Russo Nyyti Saarimäki Federica Sarro Davide Taibi Janet Siegmund Diomidis Spinellis Miroslaw Staron Klaas Stol Margaret-Anne Storey Davide Taibi Damian Tamburri Marco Torchiano Christoph Treude Burak Turhan Xiaofeng Wang and Sira Vegas. 2021. Empirical Standards for Software Engineering Research. http://arxiv.org/abs/2010.03525 Version 0.2.0. Paul Ralph Nauman bin Ali Sebastian Baltes Domenico Bianculli Jessica Diaz Yvonne Dittrich Neil Ernst Michael Felderer Robert Feldt Antonio Filieri Breno Bernard Nicolau de França Carlo Alberto Furia Greg Gay Nicolas Gold Daniel Graziotin Pinjia He Rashina Hoda Natalia Juristo Barbara Kitchenham Valentina Lenarduzzi Jorge Martínez Jorge Melegati Daniel Mendez Tim Menzies Jefferson Molleri Dietmar Pfahl Romain Robbes Daniel Russo Nyyti Saarimäki Federica Sarro Davide Taibi Janet Siegmund Diomidis Spinellis Miroslaw Staron Klaas Stol Margaret-Anne Storey Davide Taibi Damian Tamburri Marco Torchiano Christoph Treude Burak Turhan Xiaofeng Wang and Sira Vegas. 2021. Empirical Standards for Software Engineering Research. http://arxiv.org/abs/2010.03525 Version 0.2.0. · arXiv
  25. 10.1109/ICSE.2003.1201189 · DOI
  26. Connie U Smith and Lloyd G Williams . 2002. Performance solutions: a practical guide to creating responsive, scalable software . Addison-Wesley . Connie U Smith and Lloyd G Williams. 2002. Performance solutions: a practical guide to creating responsive, scalable software. Addison-Wesley.
  27. Walter F. Tichy . 1998. Should computer scientists experiment more?Computer 31, 5 (May 1998 ), 32–40. https://doi.org/10.1109/2.675631 10.1109/2.675631 Walter F. Tichy. 1998. Should computer scientists experiment more?Computer 31, 5 (May 1998), 32–40. https://doi.org/10.1109/2.675631 · DOI
  28. 10.1145/2590528.2590529 · DOI
  29. 10.1145/2668930.2688819 · DOI
  30. André van Hoorn , Matthias Rohr , and Wilhelm Hasselbring . 2008. Generating Probabilistic and Intensity-Varying Workload for Web-Based Software Systems . In Performance Evaluation: Metrics, Models and Benchmarks . Springer , 124–143. https://doi.org/10.1007/978-3-540-69814-2_9 10.1007/978-3-540-69814-2_9 André van Hoorn, Matthias Rohr, and Wilhelm Hasselbring. 2008. Generating Probabilistic and Intensity-Varying Workload for Web-Based Software Systems. In Performance Evaluation: Metrics, Models and Benchmarks. Springer, 124–143. https://doi.org/10.1007/978-3-540-69814-2_9 · DOI
  31. 10.4108/icst.valuetools.2014.258171 · DOI
  32. 10.1145/2188286.2188326 · DOI
  33. Marco Vieira , Henrique Madeira , Kai Sachs , and Samuel Kounev . 2012. Resilience Benchmarking . In Resilience Assessment and Evaluation of Computing Systems . Springer , 283–301. https://doi.org/10.1007/978-3-642-29032-9_14 10.1007/978-3-642-29032-9_14 Marco Vieira, Henrique Madeira, Kai Sachs, and Samuel Kounev. 2012. Resilience Benchmarking. In Resilience Assessment and Evaluation of Computing Systems. Springer, 283–301. https://doi.org/10.1007/978-3-642-29032-9_14 · DOI
  34. 10.1007/s10270-016-0566-5 · DOI
  35. Joakim von Kistowski , Simon Eismann , Norbert Schmitt , Andre Bauer , Johannes Grohmann , and Samuel Kounev . 2018 . TeaStore: A Micro-Service Reference Application for Benchmarking , Modeling and Resource Management Research. In 2018 IEEE 26th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS). IEEE, 223–236 . https://doi.org/10.1109/mascots.2018.00030 10.1109/mascots.2018.00030 Joakim von Kistowski, Simon Eismann, Norbert Schmitt, Andre Bauer, Johannes Grohmann, and Samuel Kounev. 2018. TeaStore: A Micro-Service Reference Application for Benchmarking, Modeling and Resource Management Research. In 2018 IEEE 26th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS). IEEE, 223–236. https://doi.org/10.1109/mascots.2018.00030 · DOI
  36. Robert von Massow , André van Hoorn , and Wilhelm Hasselbring . 2011 . Performance Simulation of Runtime Reconfigurable Component-Based Software Architectures. In Software Architecture (Proceedings ECSA 2011) (Lecture Notes in Computer Science, Vol. 6903), Ivica Crnkovic, Volker Gruhn, and Matthias Book (Eds.). Springer-Verlag, 43–58. https://doi.org/10.1007/978-3-642-23798-0_5 10.1007/978-3-642-23798-0_5 Robert von Massow, André van Hoorn, and Wilhelm Hasselbring. 2011. Performance Simulation of Runtime Reconfigurable Component-Based Software Architectures. In Software Architecture (Proceedings ECSA 2011)(Lecture Notes in Computer Science, Vol. 6903), Ivica Crnkovic, Volker Gruhn, and Matthias Book (Eds.). Springer-Verlag, 43–58. https://doi.org/10.1007/978-3-642-23798-0_5 · DOI
  37. 10.1145/2735399.2735416 · DOI
  38. 10.1007/978-3-642-31202-1_5 · DOI
  39. Claes Wohlin . 2021. Case Study Research in Software Engineering—It is a Case, and it is a Study, but is it a Case Study?Information and Software Technology 133 (May 2021 ). https://doi.org/10.1016/j.infsof.2021.106514 10.1016/j.infsof.2021.106514 Claes Wohlin. 2021. Case Study Research in Software Engineering—It is a Case, and it is a Study, but is it a Case Study?Information and Software Technology 133 (May 2021). https://doi.org/10.1016/j.infsof.2021.106514 · DOI

Cited by

Sorted by citations

Metadata

Authors
Hasselbring, W.
Affiliations
Christian-Albrechts-Universität zu Kiel (DE)
Corresponding author
Hasselbring, Wilhelm
ORCID
Hasselbring, Wilhelm
Venue
Evaluation and Assessment in Software Engineering
Publisher
ACM
Published
2021-06-21
Pages
365-372
DOI
10.1145/3463274.3463361
OpenAlex
W3158429528
Cited by
40
References
0
Citations by year
2026: 6 · 2025: 17 · 2024: 4 · 2023: 6 · 2022: 7
FWCI
2.86
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
  • arXiv (Cornell University) (acceptedVersion)
  • ePrints Soton (University of Southampton) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
Keywords
benchmarking, empirical software engineering, benchmarking standards, software performance evaluation, scalability analysis, method comparison

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review