Benchmarking scalability of stream processing frameworks deployed as microservices in the cloud

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Hazelcast Jet is a stream processing framework built on top of the Hazelcast IMDG distributed, in-memory object store.

“Hazelcast Jet (Gencer et al., 2021) is a stream processing framework built on top of the Hazelcast IMDG distributed, in-memory object store.”

• cites Gencer et al., 2021 sourceSupported

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Apache Storm and Apache Heron provide less sophisticated programming models and weaker fault tolerance mechanisms.

“Storm and Heron provide less sophisticated programming models and weaker fault tolerance mechanisms (Fragkoulis et al., 2023).”

• cites Fragkoulis et al., 2023 sourcePartly supported

  Section 4.2.1 says systems such as Storm left state management to programmers, and Section 5.5 contrasts the approximate results of earlier systems with modern systems’ exactly-once state semantics under failure. These passages support a qualified comparison for Storm, but the copy does not address Heron or directly compare the programming models and fault-tolerance mechanisms of both named systems. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Apache Flink offers one of the most elaborated dataflow models, providing precise control of time and state.

“It offers one of the most elaborated dataflow models, providing precise control of time and state (Carbone et al., 2015(Carbone et al., , 2017;;Akidau et al., 2021).”

• cites Carbone et al., 2015Not checked — no full text of the source
• cites Carbone et al., 2017 sourceNot checked — no full text of the source
• cites Akidau et al., 2021 sourcePartly supported

  Section 5.2 describes Flink as a dataflow processor with stateful operators and event-time processing; Section 5.2.3 describes timer APIs whose callbacks can modify state. These passages support control over time and state, but the paper does not establish that Flink has one of the most elaborated dataflow models. (LLM, conf=0.83)

  Experimental — accuracy not yet measured

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Apache Kafka Streams has a restricted set of features compared to most other stream processing frameworks, particularly concerning integration with external systems.

“Compared to most other stream processing frameworks, it has a restricted set of features, in particular, concerning the integration with external systems.”

• cites Sax et al., 2018 sourceCould not verify

  Section 5 describes Kafka Streams’ processing features and its use of Kafka topics, but does not compare its feature set with other frameworks. A search for external-system integration and comparative limitations found no passage addressing the claim. (LLM, conf=0.73)

  Experimental — accuracy not yet measured
• cites Wang et al., 2021 sourceCould not verify

  Section 3.2 describes Kafka Streams reading from and writing to Kafka topics, and §6.1.1 describes Kafka Connect moving data between Kafka and external systems. Neither passage compares Kafka Streams’ feature set with most other frameworks or says its external-system integration is restricted. Searches for feature limitations and external-system integration found no such comparison in our copy. (LLM, conf=0.72)

  Experimental — accuracy not yet measured

Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.

Benchmarking is a well-established method in software engineering to assess and compare the quality of software systems and services.

“Benchmarking is a wellestablished method in software engineering to assess and compare the quality of software systems and services (Hasselbring, 2021).”

• cites Hasselbring, 2021 sourceSupported

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Apache Spark, Apache Storm, and Apache Heron are considered heavyweight frameworks and fit less into the context of microservices.

“Apache Spark (Zaharia et al., 2016), Apache Storm (Toshniwal et al., 2014), and the successor of the latter, Apache Heron (Kulkarni et al., 2015), are considered heavyweight frameworks and, thus, fit less into the context of microservices (Bellemare, 2020).”

• cites Zaharia et al., 2016 sourceNot checked — no full text of the source
• cites Toshniwal et al., 2014 sourceNot checked — no full text of the source
• cites Kulkarni et al., 2015 sourceSupported

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Kubernetes is the de-facto standard orchestration tool for cloud-native applications.

“with Kubernetes (Burns et al., 2016) being the de-facto standard orchestration tool for cloud-native applications (Cloud Native Computing Foundation, 2022).”

• cites Burns et al., 2016 sourceSupported
• cites Cloud Native Computing Foundation, 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Hazelcast Jet differs from other frameworks in its execution model, which is based on a concept similar to coroutines and cooperative threads.

“Hazelcast Jet differs from other frameworks in its execution model, which is based on a concept similar to coroutines and cooperative threads (Gencer et al., 2021).”

• cites Gencer et al., 2021 sourceSupported

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Apache Samza is sometimes considered a predecessor of Kafka Streams.

“Although still maintained, Samza is sometimes considered a predecessor of Kafka Streams (Kleppmann and Kreps, 2015).”

• cites Kleppmann and Kreps, 2015 sourcePartly supported

  Section 1.4 says Samza was already in use while Kafka Streams was being added, and describes their stream-processing capabilities as similar. This supports the chronological comparison, but the paper does not say Samza was considered a predecessor of Kafka Streams. (LLM, conf=0.78)

  Experimental — accuracy not yet measured

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Modern stream processing frameworks are designed to run in a distributed fashion on commodity hardware in order to scale with massive amounts of data.

“Modern stream processing frameworks are designed to run in a distributed fashion on commodity hardware in order to scale with massive amounts of data (Fragkoulis et al., 2023).”

• cites Fragkoulis et al., 2023 sourcePartly supported

  The Introduction says stream processors execute in parallel on distributed clusters and are converging toward scale-out processing of massive streams. It does not specify that those clusters use commodity hardware, so that part of the claim is unsupported. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

The previous study applies the Theodolite method but benchmarks scalability with respect to costs and compares stream processing deployments against Function-as-a-Service offerings.

“Our previous study (Pfandzelter et al., 2022) applies our Theodolite method as well, but benchmarks scalability with respect to costs and is addressed to comparing stream processing deployments against Function-as-a-Service offerings.”

• cites Pfandzelter et al., 2022 sourcePartly supported

  The Benchmark Methodology and Cost Measurement sections describe using Theodolite components to estimate costs at different workload levels, and the Conclusions compare distributed stream processing with FaaS. However, the Benchmark Methodology explicitly says the goal is not to measure platform scalability or elasticity, so describing the study as benchmarking scalability overstates its stated aim. The passages provide no page numbers. (LLM, conf=0.85)

  Experimental — accuracy not yet measured

Probably not a substantial claim. The paper’s conclusions may not depend on it.

In practice, one can typically observe a combination of different communication methods in microservice architectures, such as additional HTTP-based communication.

“In practice, one can typically observe a combination of different communication methods, such as additional HTTP-based communication(Karabey Aksakalli et al.,”

• cites Karabey Aksakalli et al., 2021 sourceCould not verify

  The passages available in our copy are bibliographic metadata and keywords, not discussion of communication practices (the supplied passages are labeled Abstract but have no page numbers). Searches for HTTP, combinations of communication methods, and synchronous or asynchronous communication found no substantive passage. The copy therefore does not establish whether microservice architectures typically combine methods, including HTTP-based communication. (LLM, conf=0.72)

  Experimental — accuracy not yet measured

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Gencer et al. (2021) implemented the Nexmark benchmark for their performance evaluation of Hazelcast Jet.

“Moreover,Gencer et al. (2021)implemented the Nexmark benchmark for their performance evaluation of Hazelcast Jet.”

• cites Gencer et al., 2021 sourceSupported

It may state two separate claims as one.

Nasiri et al. (2019) conduct independent evaluations of scaling load and computing resources and thus address another aspect than this study.

“Nasiri et al. (2019) conduct independent evaluations of scaling load and computing resources and, thus, address another aspect than our study.”

• cites Nasiri et al. (2019) sourceSupported

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Other recent studies benchmark qualities such as throughput, latency, and fault-tolerance of stream processing systems.

“there are other recent studies that benchmark qualities such as throughput (Bordin et al., 2020;Chu et al., 2020;van Dongen and van den Poel, 2020), latency (Bordin et al., 2020;van Dongen and van den Poel, 2020;Hesse et al., 2021), and fault-tolerance (van Dongen and van den Poel, 2021b).”

• cites Bordin et al., 2020 sourceSupported
• cites Chu et al., 2020 sourceSupported
• cites van Dongen and van den Poel, 2020 sourceNot checked — no full text of the source

It may state two separate claims as one.

Vikash et al. (2020), Nasiri et al. (2019), Karakaya et al. (2017), and van Dongen and van den Poel (2021a) explicitly evaluate scalability, however without testing different load intensities against different resource amounts in isolated experiments.

“Vikash et al. (2020), Nasiri et al. (2019), Karakaya et al. (2017), and van Dongen and van den Poel (2021a) explicitly evaluate scalability, however, without testing different load intensities against different resource amounts in isolated experiments.”

• cites Vikash et al. (2020) sourceNot checked — no full text of the source
• cites Nasiri et al. (2019) sourceSupported
• cites Karakaya et al. (2017) sourceNot checked — no full text of the source

It may state two separate claims as one.

Under high load, consumer lag may not substantially increase, but records are discarded due to lateness.

“In certain cases, we observed that under high load the consumer lag does not substantially increase, but records were discarded due to lateness.”

It may state two separate claims as one.

The observed scalability results hold regardless of whether load is scaled on a microservice, computational work inside the microservice is scaled, or the selected cloud environment is changed.

“We observe our results regardless of scaling load on a microservice, scaling the computational work performed inside the microservice, and the selected cloud environment.”

• AssumptionRests on an assumption: Private and public clouds are representative environments for operating microservice-based software systems.
• AssumptionRests on an assumption: All modern stream processing frameworks can be deployed containerized on commodity hardware with Kubernetes.
• AssumptionRests on an assumption: The benchmark task samples represent typical use cases for analyzing Industrial Internet of Things sensor data streams.
• AssumptionRests on an assumption: The selected stream processing frameworks are state-of-the-art and particularly suited to be deployed as microservices.

It may state two separate claims as one.

In UC3, Flink requires up to 10 TaskManagers and Kafka Streams up to 18 instances.

“On the other hand, Flink requires up to 10 TaskManagers and Kafka Streams up to 18 instances.”

It may state two separate claims as one.

For Beam UC4, Flink's resource demand increases at a steeper rate, and for lower load intensities (less than 5 nested groups), Flink requires fewer instances than Samza.

“For benchmark UC4 (see Fig. 5(b)), it appears that the resource demand of Flink increases at a steeper rate as well.For lower load intensities (less than 5 nested groups), Flink requires fewer instances than Samza.”

It may state two separate claims as one.

In UC2, Hazelcast Jet shows the best results, followed by Kafka Streams and Flink.

“For UC2 (see Fig. 4(b)), we see a clear ranking with Hazelcast Jet showing the best results, followed by Kafka Streams and Flink.”

It may state two separate claims as one.

For Beam UC3, both frameworks scale at similar rates, with Samza requiring slightly fewer instances.

“For benchmark UC3 (see Fig. 5(a)), both frameworks scale at similar rates with Samza requiring slightly fewer instances.”

It may state two separate claims as one.

In UC4, Hazelcast Jet shows significantly higher resource demand, and up to 30 instances are not able to handle load from more than 7 nested sensor groups.

“In contrast to the other benchmarks, Hazelcast Jet shows a significantly higher resource demand.Up to 30 instances are not able to handle load from more than 7 nested sensor groups.”

It may state two separate claims as one.

The Kafka Streams implementation of benchmark UC3 scales with neither increasing the number of instances nor increasing the number of cores.

“Moreover, the Kafka Streams implementation of benchmark UC3 scales with neither increasing the number of instances nor with increasing the number of cores.”

It may state two separate claims as one.

In UC1, all frameworks behave similarly, with resource demands increasing slightly steeper for Hazelcast Jet compared to Kafka Streams and for Kafka Streams compared to Flink.

“In UC1 (see Fig. 4(a)), all frameworks behave similarly, with resource demands increasing slightly steeper for Hazelcast Jet compared to Kafka Streams and for Kafka Streams compared to Flink.”

It may state two separate claims as one.

There is no clear superior framework among the benchmarked frameworks; their ranking depends on the use case.

“There is no clear superior framework, but the ranking of the frameworks depends on the use case.”

• AssumptionRests on an assumption: Private and public clouds are representative environments for operating microservice-based software systems.
• AssumptionRests on an assumption: All modern stream processing frameworks can be deployed containerized on commodity hardware with Kubernetes.
• AssumptionRests on an assumption: The benchmark task samples represent typical use cases for analyzing Industrial Internet of Things sensor data streams.
• AssumptionRests on an assumption: The selected stream processing frameworks are state-of-the-art and particularly suited to be deployed as microservices.

It may state two separate claims as one.

For Flink and Hazelcast Jet, scaling with the number of cores is more efficient for the more complex dataflows in UC3 and UC4, while with UC2 both types of scaling yield similar results.

“For Flink and Hazelcast Jet, scaling with the number of cores is more efficient for the more complex dataflows in UC3 and UC4, while with benchmark UC2 both types of scaling yield similar results.”

It may state two separate claims as one.

For 6 instances, the number of queued messages will steadily increase over time; providing 7 instances leads to a decrease in queued messages after the warm-up period; 8 instances yields an almost constant trend line.

“However, computing a trend line (without considering the measurements from an initial warmup period), reveals that for 6 instances, the number of queued messages will steadily increase over time.Providing 7 instances leads to a decrease in queued messages after the warmup period, while 8 instances yields an almost constant trend line.”

It may state two separate claims as one.

Despite some outliers, all frameworks show linear scalability, yet with different rates.

“Despite some outliers (see Beam/Flink in UC1 and Beam/Samza in UC4), all frameworks show linear scalability, yet with different rates.”

It may state two separate claims as one.

In UC3, Hazelcast Jet appears even more superior, and a single Jet instance is sufficient for all evaluated load intensities.

“For UC3 (see Fig. 4(c)), Hazelcast Jet appears to be even more superior.A single Jet instance is sufficient for all evaluated load intensities.”

It may state two separate claims as one.

In almost all experiments, the resource demand of Apache Beam with Flink is at least twice as high as native Apache Flink, and for UC3 and UC4 it is tremendously higher.

“In almost all experiments, the resource demand of Apache Beam with Flink is at least twice as high.For the more compute-intensive benchmarks UC3 and UC4, it is tremendously higher.”

It may state two separate claims as one.

Disabling all logging only has a small impact on the resource demand of UC2-UC4, but significantly reduces the resource demand of UC1.

“We can observe that disabling all logging only has a small impact on the resource demand of benchmark UC2-UC4 (see Figs. 7(b)-7(d)), but significantly reduces the resource demand of benchmark UC1 (see Fig. 7(a)).”

It may state two separate claims as one.

All evaluated stream processing frameworks (Apache Flink, Apache Kafka Streams, Hazelcast Jet, and Apache Beam with the Flink and Samza runners) show linear scalability for most use cases, however with partially significantly different resource demands.

“We find that all frameworks show linear scalability for most use cases, however with partially significantly different resource demands.”

It may state two separate claims as one.

For the exemplary benchmark execution, providing 7 instances leads to a decrease in queued messages after the warmup period, while 8 instances yields an almost constant trend line.

“Providing 7 instances leads to a decrease in queued messages after the warmup period, while 8 instances yields an almost constant trend line.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

From Table 2, a lot of open-source benchmarks have been proposed.

“From Table 2, we can see that a lot of open-source benchmarks have been proposed.”

It may state two separate claims as one.

The observed scalability results hold regardless of whether the load on a microservice is scaled, the computational work performed inside the microservice is scaled, the selected cloud environment (private cloud or Google cloud), and whether the microservice is scaled over multiple nodes or on a single node.

“We observe our results regardless of scaling load on a microservice, scaling the computational work performed inside the microservice, the selected cloud environment, and whether the microservice is scaled over multiple nodes or on a single node.”

It may state two separate claims as one.

The results for the private and public clouds are similar, and all frameworks show linear scalability independent of the cloud platform.

“In general, we can observe that the results for both clouds are similar and all frameworks show linear scalability, independent of the cloud platform.”

It may state two separate claims as one.

Because Kafka Streams UC3 scales linearly on multiple nodes, the results suggest that underlying hardware resources become exhausted.

“As it scales linearly when running on multiple nodes (see Fig. 5(a)), our results suggest that underlying hardware resources become exhausted.”

It may state two separate claims as one.

While scalable microservices can be designed with all evaluated frameworks, the choice of framework and its deployment has a considerable impact on the cost of operating it.

“While scalable microservices can be designed with all evaluated frameworks, the choice of a framework and its deployment has a considerable impact on the cost of operating it.”

• AssumptionRests on an assumption: Private and public clouds are representative environments for operating microservice-based software systems.
• AssumptionRests on an assumption: All modern stream processing frameworks can be deployed containerized on commodity hardware with Kubernetes.
• AssumptionRests on an assumption: The benchmark task samples represent typical use cases for analyzing Industrial Internet of Things sensor data streams.
• AssumptionRests on an assumption: The selected stream processing frameworks are state-of-the-art and particularly suited to be deployed as microservices.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The performance overhead of using Apache Beam as an abstraction layer has also been observed in related research.

“The performance overhead of using Apache Beam as an abstraction layer has also been observed in related research (Hesse et al., 2019).”

• cites Hesse et al., 2019 sourceNot checked — no full text of the source

Probably not a substantial claim.

The choice of a stream processing framework is likely to have a crucial impact on the scalability of the overall system.

“The choice of a stream processing framework is thus likely to have a crucial impact on the scalability of the overall system.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Apache Beam implements Google's Dataflow model.

“Apache Beam implements Google's Dataflow model (Akidau et al., 2015), which is also internally used by Google's cloud service Google Cloud Dataflow.”

• cites Akidau et al., 2015 sourceNot checked — no full text of the source

Probably not a substantial claim.

For a fair comparison, all benchmarked frameworks are evaluated with mostly their default configuration.

“For a fair comparison, we evaluate all frameworks with mostly their default configuration.”

Probably not a substantial claim. The authors may not put it forward as true.

The previously discovered negative impact on performance of using Apache Beam as an abstraction layer persists.

“RQ 2: The previously discovered negative impact on performance of using Apache Beam as abstraction layer persists.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Apache Samza can be embedded as a library in standalone applications.

“Similar to Kafka Streams, Apache Samza (Noghabi et al., 2017) can be embedded as a library in standalone applications.”

• cites Noghabi et al., 2017 sourceNot checked — no full text of the source

Probably not a substantial claim. The paper’s conclusions may not depend on it.

A possible explanation for the lack of FasterCopy effect in UC4 is that the dataflow architecture of UC4 involves more data transfer among instances and requires serialization and deserialization between operators.

“A possible explanation is that the dataflow architecture of UC4 involves more data transfer among instances and, hence, actually requires serialization and deserialization between operators.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Zhang et al. (2020) report that they tremendously improved the performance of Beam's Samza runner primarily by exporting Beam metrics more efficiently, and that disabling Beam metrics further improved performance.

“In a blog post, software engineers at LinkedIn Zhang et al. (2020) report how they tremendously improved the performance of Beam's Samza runner.Primarily, this was achieved by exporting Beam metrics more efficiently.Moreover, the authors observed that performance could further be improved when disabling the Beam metrics entirely.”

• cites Zhang et al. (2020)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

These task samples are the only benchmarks focusing on modern stream processing frameworks deployed as cloud-native microservices.

“As we discuss in Section 8, these are the only benchmarks focusing on modern stream processing frameworks deployed as cloud-native microservices.”

It may state two separate claims as one.

Despite some exceptions for specific task samples, stream processing frameworks allow scaling on a single node, making vertical scaling a complementary measure to achieve scalability.

“RQ 4: Despite some exceptions for specific task samples, stream processing frameworks allow scaling them on a single node, making vertical scaling a complementary measure to achieve scalability of stream processing frameworks.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The Theodolite scalability benchmarking method provides two alternative scalability metrics: the resource demand metric and the load capacity metric.

“The Theodolite scalability benchmarking method provides two alternative scalability metrics, the resource demand metric and the load capacity metric.”

• cites Henning and Hasselbring, 2022aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Apache Kafka Streams only supports Kafka topics as data sources and sinks.

“Kafka Streams only supports Kafka topics as data sources and sinks.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Apache Beam is not a stream processing system by itself, but an SDK to implement stream processing jobs in a uniform model.

“Apache Beam is not a stream processing system by itself, but instead, an SDK to implement stream processing jobs in a uniform model, which can be executed by several modern stream processing systems.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Previous research found that using Apache Beam as an abstraction layer comes with a significant negative impact on performance.

“Previous research found that using Apache Beam as an abstraction layer comes with a significant negative impact on performance (Hesse et al., 2019).”

• cites Hesse et al., 2019 sourceNot checked — no full text of the source

Probably not a substantial claim.

The authors' study builds upon their Theodolite scalability benchmarking method for cloud-native applications.

“Our study builds upon our Theodolite scalability benchmarking method for cloud-native applications (Henning and Hasselbring, 2022a).”

• cites Henning and Hasselbring, 2022aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. It may state two separate claims as one.

Spæren (2021) discovered unnecessary serialization and deserialization between operators and introduced the FasterCopy option, which disables these copy operations.

“They discovered unnecessary serialization and deserialization between operators and introduced the FasterCopy option, which disables these copy operations.”

• cites Spæren (2021)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Hesse et al. (2019) sourceNot checked — no full text of the source

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Apache Spark processes data streams in micro-batches.

“Spark differs from the frameworks discussed before in that it processes data streams in ''micro-batches''.”

• cites Zaharia et al., 2016 sourceNot checked — no full text of the source

Probably not a substantial claim.

In previous work, the authors presented and empirically evaluated a scalability benchmarking method for cloud-native applications and stream processing frameworks.

“In previous work, we presented and empirically evaluated a scalability benchmarking method for cloud-native applications in general (Henning and Hasselbring, 2022a) and stream processing frameworks in particular (Henning and Hasselbring, 2021a).”

• cites Henning and Hasselbring, 2022aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Henning and Hasselbring, 2021aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

For the Apache Beam implementations with the Samza and Flink runners, the authors cannot definitely be sure whether the determined resource demand for UC2 and UC4 is sufficient to process all records successfully, but it still indicates a lower bound.

“This means that for these two SUTs, we cannot definitely be sure whether the determined resource demand for UC2 and UC4 is sufficient to process all records successfully, but still indicates a lower bound.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The authors repeat all experiments 3 times, which they experimentally observed to be a good trade-off between overall execution time or cost and statistical grounding.

“We repeat all our experiments 3 times, which we experimentally observed to be a good trade-off between overall execution time (or costs in the public cloud) and statistical grounding (Henning and Hasselbring, 2022a).”

• cites Henning and Hasselbring, 2022aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Based on real-world use cases for Industrial Internet of Things analytics, the authors proposed four task samples of different complexity for stream processing frameworks in previous work.

“Based on real-world use cases for Industrial Internet of Things analytics (Henning et al., 2021), we proposed four task samples of different complexity for stream processing frameworks in previous work (Henning and Hasselbring, 2021b).”

• cites Henning et al., 2021Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Henning and Hasselbring, 2021bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

The increase in resource demand differs noticeably between frameworks, with Hazelcast Jet superior for the studied case, followed by Flink and Kafka Streams, and significantly higher resource demand of the Beam-based deployments.

“Again, however, the increase in resource demand differs noticeably between frameworks with Hazelcast Jet being superior for the studied case, followed by Flink and Kafka Streams, and significantly higher resource demand of the Beam-based deployments.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Apache Flink falls into the category of heavyweight stream processing frameworks.

“Flink clearly falls into the category of heavyweight frameworks (Bellemare, 2020).”

• cites Bellemare, 2020Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

All frameworks appear to be linearly scalable, however with different resource usage.

“RQ 1: All frameworks appear to be linearly scalable, however, with different resource usage.”

Probably not a substantial claim.

The FasterCopy option is integrated in Beam since version 2.26.

“This option is integrated in Beam since version 2.26.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

9 out of 20 studies report on experiments in public or private clouds.

“9 out of 20 studies report on experiments in public or private clouds.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The Apache Beam community adapted and extended Nexmark with implementations for Beam to benchmark the performance of different runners.

“The Apache Beam community adapted and extended Nexmark with implementations for Beam to benchmark the performance of different runners (Apache Software Foundation, 2022).”

• cites Apache Software Foundation, 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

Although the evaluations in Section 6.5 show only small deviations between the private and Google Cloud environments, the same results cannot necessarily be concluded for other cloud environments.

“Although our evaluations in Section 6.5 show only small deviations between both environments, we cannot necessarily conclude that we would obtain the same results in other cloud environments.”

It may state two separate claims as one.

Kafka Streams, Hazelcast Jet, and Samza are only benchmarked in a few studies, and there is no study benchmarking all of them.

“Kafka Stream, Hazelcast Jet, and Samza, which are particularly suited to be deployed as microservices, are only benchmarked in a few studies and there is no study benchmarking all of them.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Flink, Spark, and Storm are by far the most supported frameworks among the discussed benchmarks.

“Flink, Spark, and Storm are by far the most supported frameworks.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Cloud-native containerized deployments as in this study further abstract the underlying hardware or software infrastructure.

“Cloud-native containerized deployments as in this study further abstract this.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

Repeating individual experiments more often and for longer time periods as well as using the full search strategy is likely to produce very similar, but not necessarily identical, results.

“Repeating individual experiments more often and for longer time periods as well as using our full search strategy is likely to produce very similar, but not necessarily identical results.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

Most benchmarks include a messaging system as a middleware component between workload generation and stream processing framework, and in the vast majority of cases this is Apache Kafka.

“Most benchmarks include a messaging system as a middleware component between workload generation and stream processing framework.In the vast majority of cases, this is Apache Kafka.”

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

The limitation of a larger evaluation set is addressed by carefully observing the benchmark execution, but the experiments are not as extensive as in the method's evaluation.

“We address this limitation by carefully observing the benchmark execution but do not conduct as extensive experiments as in our method's evaluation.”

It may state two separate claims as one.

While scalable microservices can be designed with all evaluated frameworks, the choice of a framework and its deployment has a considerable impact on the cost of operating it.

“In summary we found that while scalable microservices can be designed with all evaluated frameworks, the choice of a framework and its deployment has a considerable impact on the cost of operating it.”

Probably not a substantial claim.

Cloud platforms in general allow only making little assumptions regarding the underlying hardware or software infrastructure.

“Cloud platforms in general allow only making little assumptions regarding the underlying hardware or software infrastructure (Bermbach et al., 2017).”

• cites Bermbach et al., 2017Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

It may state two separate claims as one.

YSB can be considered a micro-benchmark and is less suited to benchmark entire microservices.

“However, YSB can be considered a micro-benchmark (Bermbach et al., 2017) and, hence, is less suited to benchmark entire microservices.”

• cites Bermbach et al., 2017Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The Theodolite benchmarks purposely include Kafka to represent more realistic microservice deployments.

“Our Theodolite benchmarks purposely include Kafka to represent more realistic microservice deployments (Henning and Hasselbring, 2021b).”

• cites Henning and Hasselbring, 2021bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

In our experiments, resource demand increases steeper in the public cloud, yet this is expected to be due to the specific machine types selected.

“In our experiments, resource demand increases steeper in the public cloud, yet we expect this to be due to the specific machine types selected.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Flink, Spark, and Storm are by far the most frequently benchmarked frameworks.

“Flink, Spark, and Strom are by far the most frequently benchmarked frameworks.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

This study evaluates a larger set of SUTs, load types, resource types, SLOs, and benchmarks compared to the method's evaluation.

“Moreover, we evaluate a larger set of SUTs, load types, resource types, SLOs, and benchmarks in this study compared to our method's evaluation.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

Depending on the microservice-specific use case, microservice architectures may also include microservices relying on other mechanisms to process data streams, such as simple producers/consumers or Function-as-a-Service deployments.

“Depending on the microservice-specific use case, such architectures may also include microservices relying on other mechanisms to process data streams, such as simple producers/consumers or Function-as-a-Service deployments(Bellemare, 2020).”

• cites Bellemare, 2020Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim.

The evaluations are built upon the results of the benchmarking method's evaluation to obtain statistically grounded benchmarking results.

“To obtain statistically grounded benchmarking results, we build these evaluations upon the results of our benchmarking method's evaluation (Henning and Hasselbring, 2022a).”

• cites Henning and Hasselbring, 2022aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim. The paper may give no reason why it holds.

Assessing frameworks primarily using default configurations should mitigate bias stemming from different levels of experience with the different frameworks.

“This should mitigate any bias stemming from different levels of our experience with the different frameworks.”

It may state two separate claims as one.

Observed scalability limits seem to be caused by utilized hardware and not by the streaming frameworks themselves, so limits can be raised by using larger clusters.

“RQ 6: Observed scalability limits seem to be caused by utilized hardware and not by the streaming frameworks themselves.Hence, limits can be raised by using larger clusters.”

Probably not a substantial claim.

All frameworks are deliberately assessed primarily using their default configurations.

“We deliberately assess all frameworks primarily using their default configurations.”

Probably not a substantial claim. The paper’s conclusions may not depend on it.

The steeper increase in resource demand in the Google cloud is expected to be due to using Google's general-purpose E2 virtual machines with potentially less powerful resources.

“We expect that this is due to the fact that we used Google's general-purpose E2 virtual machines with potentially less powerful resources.”

Probably not a substantial claim.

A key benefit of designing stream processing applications using microservice-based architectures is the ability for each service to select the most suitable stream framework based on its specific use case, requirements for scalability, and other relevant considerations.

“This emphasizes also a key benefit of designing stream processing applications using microservice-based architectures: the ability for each service to select the most suitable stream framework based on its specific use case, requirements for scalability, and other relevant considerations.”