CoChat
Sign Up
Cited by 27Advanced Database Systems and QueriesParallel Computing and Optimization TechniquesCloud Computing and Resource Management

The DOI matches its Crossref record.

ESPBench: The Enterprise Stream Processing Benchmark

Open access
Hesse, Guenter, Matthies, Christoph, Perscheid, Michael, Uflacker, Matthias, Plattner, Hasso·2021·Proceedings of the ACM/SPEC International Conference on Performance Engineering
DOI 10.1145/3427921.3450242
Open this paper in CoChat — free accountOpen PDF

Abstract

Growing data volumes and velocities in fields such as Industry 4.0 or the Internet of Things have led to the increased popularity of data stream processing systems. Enterprises can leverage these developments by enriching their core business data and analyses with up-to-date streaming data. Comparing streaming architectures for these complex use cases is challenging, as existing benchmarks do not cover them. ESPBench is a new enterprise stream processing benchmark that fills this gap. We present its architecture, the benchmarking process, and the query workload. We employ ESPBench on three state-of-the-art stream processing systems, Apache Spark, Apache Flink, and Hazelcast Jet, using provided query implementations developed with Apache Beam. Our results highlight the need for the provided ESPBench toolkit that supports benchmark execution, as it enables query result validation and objective latency measures.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3427921.3450242

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • There is only little empirical research evaluating and comparing the scalability of open-source stream processing frameworks.

    “While there are several open-source stream processing frameworks promoting scalability as a core feature, there is only little empirical research evaluating and comparing their scalability.”

    From Benchmarking scalability of stream processing frameworks deployed as microservices in the cloud (2024) · cites this paper as Hesse et al., 2021

    The Related Work section (page unavailable) says that only a few benchmarks focus on stream-processing architectures, supporting a general shortage of benchmarking work. It does not establish that empirical research specifically comparing the scalability of open-source frameworks is scarce. The Conclusion and Future Work section (page unavailable) instead identifies analysis of those systems’ scalability characteristics as future work. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 33

  1. 10.1007/s00778-004-0147-z · DOI
  2. 10.1016/B978-012088469-8.50044-9 · DOI
  3. 10.14778/1920841.1920874 · DOI
  4. Paris Carbone , Asterios Katsifodimos , Stephan Ewen , Volker Markl , Seif Haridi ,and Kostas Tzoumas . 2015 . Apache Flink: Stream and Batch Processing in a Single Engine . IEEE Data Eng. Bull. 38 , 4 (2015), 28 -- 38 . Paris Carbone, Asterios Katsifodimos, Stephan Ewen, Volker Markl, Seif Haridi,and Kostas Tzoumas. 2015. Apache Flink: Stream and Batch Processing in a Single Engine. IEEE Data Eng. Bull.38, 4 (2015), 28--38.
  5. Sanket Chintapalli , Derek Dagit , Bobby Evans , Reza Farivar , Thomas Graves , Mark Holderbaugh , Zhuo Liu , Kyle Nusbaum , Kishorkumar Patil , Boyang Peng ,and Paul Poulosky . 2016 . Benchmarking Streaming Computation Engines: Storm , Flink and Spark Streaming. In IEEE International Parallel and Distributed Processing Symposium (IPDPS) Workshops. 1789--1792 . Sanket Chintapalli, Derek Dagit, Bobby Evans, Reza Farivar, Thomas Graves, Mark Holderbaugh, Zhuo Liu, Kyle Nusbaum, Kishorkumar Patil, Boyang Peng,and Paul Poulosky. 2016. Benchmarking Streaming Computation Engines: Storm, Flink and Spark Streaming. In IEEE International Parallel and Distributed Processing Symposium (IPDPS) Workshops. 1789--1792.
  6. Marlon Dumas and Arthur H. M . ter Hofstede. 2001. UML Activity Diagrams asa Workflow Specification Language . In UML 2001 - The Unified Modeling Language, Modeling Languages, Concepts, and Tools. 76--90 . Marlon Dumas and Arthur H. M. ter Hofstede. 2001. UML Activity Diagrams asa Workflow Specification Language. In UML 2001 - The Unified Modeling Language, Modeling Languages, Concepts, and Tools. 76--90.
  7. Jim Gray (Ed.). 1993. The Benchmark Handbook for Database and Transaction Processing Systems (2nd Edition). Morgan Kaufmann. Jim Gray (Ed.). 1993. The Benchmark Handbook for Database and Transaction Processing Systems (2nd Edition). Morgan Kaufmann.
  8. Guenter Hesse and Martin Lorenz . 2015 . Conceptual Survey on Data Stream Processing Systems. In IEEE International Conference on Parallel and Distributed Systems, ICPADS. 797--802 . Guenter Hesse and Martin Lorenz. 2015. Conceptual Survey on Data Stream Processing Systems. In IEEE International Conference on Parallel and Distributed Systems, ICPADS. 797--802.
  9. 10.1109/ICDCS.2019.00138 · DOI · Paper page
  10. 10.1145/3093742.3093902 · DOI
  11. 10.1007/978-3-030-18590-9_80 · DOI
  12. Guenter Hesse , Christoph Matthies , and Matthias Uflacker . 2020 . How Fast Can We Insert? A Performance Study of Apache Kafka . In IEEE International Conference on Parallel and Distributed Systems, ICPADS. Guenter Hesse, Christoph Matthies, and Matthias Uflacker. 2020. How Fast Can We Insert? A Performance Study of Apache Kafka. In IEEE International Conference on Parallel and Distributed Systems, ICPADS.
  13. Guenter Hesse , Benjamin Reissaus , Christoph Matthies , Martin Lorenz , Milena Kraus , and Matthias Uflacker . 2017 . Senska - Towards an Enterprise Streaming Benchmark . In TPC Technology Conference, TPCTC. 25--40 . Guenter Hesse, Benjamin Reissaus, Christoph Matthies, Martin Lorenz, Milena Kraus, and Matthias Uflacker. 2017. Senska - Towards an Enterprise Streaming Benchmark. In TPC Technology Conference, TPCTC. 25--40.
  14. 10.5220/0007950203040310 · DOI
  15. Marco F. Huber Martin Voigt and Axel-Cyrille Ngonga Ngomo. 2016. Big Data Architecture for the Semantic Analysis of Complex Events in Manufacturing. In 46. Jahrestagung der Gesellschaft für Informatik. 353--360. http://subs.emis.de/LNI/Proceedings/Proceedings259/article173.html Marco F. Huber Martin Voigt and Axel-Cyrille Ngonga Ngomo. 2016. Big Data Architecture for the Semantic Analysis of Complex Events in Manufacturing. In 46. Jahrestagung der Gesellschaft für Informatik. 353--360. http://subs.emis.de/LNI/Proceedings/Proceedings259/article173.html
  16. 10.1007/978-3-642-10424-4_3 · DOI
  17. J.H.M. Janssens . 2013. Outlier Selection and One-Class Classification . Ph.D. Dissertation . Series: TiCC Ph.D . Series Volume: 27 . J.H.M. Janssens. 2013.Outlier Selection and One-Class Classification. Ph.D. Dissertation. Series: TiCC Ph.D. Series Volume: 27.
  18. Zbigniew Jerzak , Thomas Heinze , Matthias Fehr , Daniel Gröber , Raik Hartung , and Nenad Stojanovic . 2012 . The DEBS 2012 Grand Challenge. In ACM International Conference on Distributed Event-Based Systems, DEBS. 393--398 . Zbigniew Jerzak, Thomas Heinze, Matthias Fehr, Daniel Gröber, Raik Hartung, and Nenad Stojanovic. 2012. The DEBS 2012 Grand Challenge. In ACM International Conference on Distributed Event-Based Systems, DEBS. 393--398.
  19. M. Johns . 2015. Getting Started with Hazelcast - Second Edition. Packt Publishing . https://books.google.de/books?id=r8cIswEACAAJ M. Johns. 2015. Getting Started with Hazelcast - Second Edition. Packt Publishing. https://books.google.de/books?id=r8cIswEACAAJ
  20. Jeyhun Karimov , Tilmann Rabl , Asterios Katsifodimos , Roman Samarev , Henri Heiskanen , and Volker Markl . 2018 . Benchmarking Distributed Stream Data Processing Systems. In IEEE International Conference on Data Engineering (ICDE). 1507--1518 . Jeyhun Karimov, Tilmann Rabl, Asterios Katsifodimos, Roman Samarev, Henri Heiskanen, and Volker Markl. 2018. Benchmarking Distributed Stream Data Processing Systems. In IEEE International Conference on Data Engineering (ICDE). 1507--1518.
  21. Jay Kreps , Neha Narkhede , and Jun Rao . 2011 . Kafka: a Distributed Messaging System for Log Processing . In International Workshop on Networking Meets Databases (NetDB). 1--7. Jay Kreps, Neha Narkhede, and Jun Rao. 2011. Kafka: a Distributed Messaging System for Log Processing. In International Workshop on Networking Meets Databases (NetDB). 1--7.
  22. Scott T. Leutenegger and Daniel M. Dias. 1993. A Modeling Study of the TPC-C Benchmark . In ACM SIGMOD International Conference on Management of Data. 22--31 . Scott T. Leutenegger and Daniel M. Dias. 1993. A Modeling Study of the TPC-C Benchmark. In ACM SIGMOD International Conference on Management of Data. 22--31.
  23. Marcelo Rodrigues Nunes Mendes. 2014.Performance Evaluation and Benchmarking of Event Processing Systems. Ph.D. Dissertation. Marcelo Rodrigues Nunes Mendes. 2014.Performance Evaluation and Benchmarking of Event Processing Systems. Ph.D. Dissertation.
  24. Marcelo R. N. Mendes , Pedro Bizarro , and Paulo Marques . 2009 . A Performance Study of Event Processing Systems. In Performance Evaluation and Benchmarking , TPC Technology Conference, TPCTC. 221--236 . Marcelo R. N. Mendes, Pedro Bizarro, and Paulo Marques. 2009. A Performance Study of Event Processing Systems. In Performance Evaluation and Benchmarking, TPC Technology Conference, TPCTC. 221--236.
  25. Judit Nagy Judit Oláh Edina Erdei Domicián Máté and József Popp. [n.d.]. The Role and Impact of Industry 4.0 and the Internet of Things on the Business Strategy of the Value Chain-The Case of Hungary. Sustainability 10 10 ([n. d.]). Judit Nagy Judit Oláh Edina Erdei Domicián Máté and József Popp. [n.d.]. The Role and Impact of Industry 4.0 and the Internet of Things on the Business Strategy of the Value Chain-The Case of Hungary. Sustainability 10 10 ([n. d.]). · DOI
  26. 10.1145/1559845.1559846 · DOI
  27. Anshu Shukla , Shilpa Chaturvedi , and Yogesh Simmhan . 2017. RIoT Bench: An IoT benchmark for distributed stream processing systems. Concurrency and Computation: Practice and Experience29, 21 ( 2017 ). Anshu Shukla, Shilpa Chaturvedi, and Yogesh Simmhan. 2017. RIoT Bench: An IoT benchmark for distributed stream processing systems. Concurrency and Computation: Practice and Experience29, 21 (2017).
  28. 10.1145/1107499.1107504 · DOI
  29. Michael Stonebraker and Lawrence A. Rowe . 1986 . THE DESIGN OF POSTGRES. In ACM SIGMOD International Conference on Management of Data. 340--355 . Michael Stonebraker and Lawrence A. Rowe. 1986. THE DESIGN OF POSTGRES. In ACM SIGMOD International Conference on Management of Data. 340--355.
  30. 10.1109/TPDS.2020.2978480 · DOI · Paper page
  31. 10.1145/2668930.2688819 · DOI
  32. Matei Zaharia , Mosharaf Chowdhury , Michael J. Franklin , Scott Shenker , and Ion Stoica . 2010 . Spark: Cluster Computing with Working Sets. In USENIX Workshop on Hot Topics in Cloud Computing, Hot Cloud. Matei Zaharia, Mosharaf Chowdhury, Michael J. Franklin, Scott Shenker, and Ion Stoica. 2010. Spark: Cluster Computing with Working Sets. In USENIX Workshop on Hot Topics in Cloud Computing, Hot Cloud.
  33. 10.1145/2517349.2522737 · DOI

Cited by

Sorted by citations

Metadata

Authors
Hesse, G., Matthies, C., Perscheid, M., Uflacker, M., & Plattner, H.
Affiliations
Hasso Plattner Institute (DE)
Venue
Proceedings of the ACM/SPEC International Conference on Performance Engineering
Publisher
ACM
Published
2021-04-09
Pages
201-212
DOI
10.1145/3427921.3450242
OpenAlex
W3135249002
Cited by
27
References
0
Citations by year
2026: 5 · 2025: 7 · 2024: 3 · 2023: 9 · 2022: 2 · 2021: 1
FWCI
2.52
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • ACM/SPEC International Conference on Performance Engineering (ICPE) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Advanced Database Systems and Queries
  • Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
  • Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
Keywords
benchmarking, query workload, Apache Spark, Apache Flink, Apache Beam, latency measurement, Industry 4.0, Internet of Things

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review