CoChat
Sign Up
Cited by 117Advanced Database Systems and QueriesCloud Computing and Resource ManagementIoT and Edge/Fog Computing

The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.

Evaluation of distributed stream processing frameworks for IoT applications in Smart Cities

Open access
Nasiri, Hamid, Nasehi, Saeed, Goudarzi, Maziar·2019·Journal of Big Data
DOI 10.1186/s40537-019-0215-2
Open this paper in CoChat — free accountOpen PDF

Abstract

The widespread growth of Big Data and the evolution of Internet of Things (IoT) technologies enable cities to obtain valuable intelligence from a large amount of real-time produced data. In a Smart City, various IoT devices generate streams of data continuously which need to be analyzed within a short period of time; using some Big Data technique. Distributed stream processing frameworks (DSPFs) have the capacity to handle real-time data processing for Smart Cities. In this paper, we examine the applicability of employing distributed stream processing frameworks at the data processing layer of Smart City and appraising the current state of their adoption and maturity among the IoT applications. Our experiments focus on evaluating the performance of three DSPFs, namely Apache Storm, Apache Spark Streaming, and Apache Flink. According to our obtained results, choosing a proper framework at the data analytics layer of a Smart City requires enough knowledge about the characteristics of target applications. Finally, we conclude each of the frameworks studied here have their advantages and disadvantages. Our experiments show Storm and Flink have very similar performance, and Spark Streaming, has much higher latency, while it provides higher throughput.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s40537-019-0215-2

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Nasiri et al. (2019) conduct independent evaluations of scaling load and computing resources and thus address another aspect than this study.

    “Nasiri et al. (2019) conduct independent evaluations of scaling load and computing resources and, thus, address another aspect than our study.”

    From Benchmarking scalability of stream processing frameworks deployed as microservices in the cloud (2024) · cites this paper as Nasiri et al. (2019)

  • Vikash et al. (2020), Nasiri et al. (2019), Karakaya et al. (2017), and van Dongen and van den Poel (2021a) explicitly evaluate scalability, however without testing different load intensities against different resource amounts in isolated experiments.

    “Vikash et al. (2020), Nasiri et al. (2019), Karakaya et al. (2017), and van Dongen and van den Poel (2021a) explicitly evaluate scalability, however, without testing different load intensities against different resource amounts in isolated experiments.”

    From Benchmarking scalability of stream processing frameworks deployed as microservices in the cloud (2024) · cites this paper as Nasiri et al. (2019)

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 29

  1. Díaz M, Martín C, Rubio B. State-of-the-art, challenges, and open issues in the integration of internet of things and cloud computing. J Netw Comput Appl. 2016;67:99–117. · DOI
  2. Zhu C, Zhou H, Leung VC, Wang K, Zhang Y, Yang LT. Toward big data in green city. IEEE Commun Mag. 2017;55(11):14–8. · DOI
  3. Chen F, Deng P, Wan J, Zhang D, Vasilakos AV, Rong X. Data mining for the internet of things: literature review and challenges. Int J Distrib Sens Netw. 2015;11(8):431047. · DOI
  4. Guo Y, Rao J, Jiang C, Zhou X. Moving hadoop into the cloud with flexible slot management and speculative execution. IEEE Trans Parallel Distrib Syst. 2017;3:798–812. · DOI
  5. Dean J, Ghemawat S. Mapreduce: simplified data processing on large clusters. Commun ACM. 2008;51(1):107–13. · DOI · Paper page
  6. Goudarzi M. Heterogeneous architectures for big data batch processing in mapreduce paradigm. IEEE Trans Big Data. 2017. https://doi.org/10.1109/TBDATA.2017.2736557 . · DOI
  7. Toshniwal A, Taneja S, Shukla A, Ramasamy K, Patel JM, Kulkarni S, Jackson J, Gade K, Fu M, Donham J, et al. Storm@twitter. In: Proceedings of the 2014 ACM SIGMOD international conference on management of data. New York: ACM; 2014. p. 147–56. · DOI · Paper page
  8. Zaharia M, Das T, Li H, Shenker S, Stoica I. Discretized streams: an efficient and fault-tolerant model for stream processing on large clusters. HotCloud. 2012;12:10.
  9. Katsifodimos A, Schelter S. Apache flink: stream analytics at scale. In: 2016 IEEE international conference on cloud engineering workshop (IC2EW). New York: IEEE; 2016. p. 193. · DOI
  10. Hashem IAT, Chang V, Anuar NB, Adewole K, Yaqoob I, Gani A, Ahmed E, Chiroma H. The role of big data in smart city. Int J Inf Manag. 2016;36(5):748–58. · DOI · Paper page
  11. Nasiri H, Nasehi S, Goudarzi M. A survey of distributed stream processing systems for smart city data analytics. In: Proceedings of the international conference on smart cities and internet of things. New York: ACM; 2018. p. 12. · DOI
  12. Hesse G, Lorenz M. Conceptual survey on data stream processing systems. In: 2015 IEEE 21st international conference on parallel and distributed systems (ICPADS). New York: IEEE; 2015. p. 797–802. · DOI
  13. Singh MP, Hoque MA, Tarkoma S. A survey of systems for massive stream analytics; 2016. arXiv preprint arXiv:1605.09021 . · arXiv
  14. Kamburugamuve S, Fox G, Leake D, Qiu J. Survey of distributed stream processing for large stream sources. 2013. https://scholar.google.com/scholar?hl=en%26as_sdt=0%2C5%26q=Survey+of+distributed+stream+processing+for+large+stream+sources%26btnG= .
  15. Kamburugamuve S, Fox G. Survey of distributed stream processing. Bloomington: Indiana University; 2016.
  16. Pääkkönen P, Pakkala D. Reference architecture and classification of technologies, products and services for big data systems. Big Data Res. 2015;2(4):166–86. · DOI
  17. Singh D, Reddy CK. A survey on platforms for big data analytics. J Big Data. 2015;2(1):8. · DOI
  18. Tsai C-W, Lai C-F, Chao H-C, Vasilakos AV. Big data analytics: a survey. J Big Data. 2015;2(1):21. · DOI · Paper page
  19. Inoubli W, Aridhi S, Mezni H, Maddouri M, Nguifo EM. An experimental survey on big data frameworks. Fut Gener Comput Syst. 2018;86:546–64. · DOI
  20. Veiga J, Expósito RR, Pardo XC, Taboada GL, Tourifio J. Performance evaluation of big data frameworks for large-scale data analytics. In: 2016 IEEE international conference on Big Data (Big Data). New York: IEEE; 2016. p. 424–31. · DOI
  21. Zaharia M, Xin RS, Wendell P, Das T, Armbrust M, Dave A, Meng X, Rosen J, Venkataraman S, Franklin MJ, et al. Apache spark: a unified engine for big data processing. Commun ACM. 2016;59(11):56–65. · DOI · Paper page
  22. Hunt P, Konar M, Junqueira FP, Reed, B. Zookeeper: wait-free coordination for internet-scale systems. In: USENIX annual technical conference, vol. 8, Boston, MA, USA; 2010.
  23. Kulkarni S, Bhagat N, Fu M, Kedigehalli V, Kellogg C, Mittal S, Patel JM, Ramasamy K, Taneja S. Twitter heron: stream processing at scale. In: Proceedings of the 2015 ACM SIGMOD international conference on management of data. New York: ACM; 2015. p. 239–50. · DOI · Paper page
  24. Vavilapalli VK, Murthy AC, Douglas C, Agarwal S, Konar M, Evans R, Graves T, Lowe J, Shah H, Seth S et al. Apache hadoop yarn: Yet another resource negotiator. In: Proceedings of the 4th annual symposium on cloud computing. New York: ACM; 2013. p. 5. · DOI · Paper page
  25. Gorawski M, Gorawska A, Pasterak K. A survey of data stream processing tools. In: Czachórski T, Gelenbe E, Lent R, editors. Information sciences and systems 2014. Cham: Springer; 2014. p. 295–303. · DOI
  26. Kejariwal A, Kulkarni S, Ramasamy K. Real time analytics: algorithms and systems. Proc VLDB Endow. 2015;8(12):2040–1. · DOI
  27. Kreps J, Narkhede N, Rao J et al. Kafka: A distributed messaging system for log processing. In: Proceedings of the NetDB; 2011. p. 1–7.
  28. Shukla A, Chaturvedi S, Simmhan Y. Riotbench: an iot benchmark for distributed stream processing systems. Concurr Comput Pract Exp. 2017;29(21):4257. · DOI
  29. Ahmadvand H, Goudarzi M, Foroutan F. Gapprox: using gallup approach for approximation in big data processing. J Big Data. 2019;6(1):20. · DOI

Cited by

Sorted by citations

Metadata

Authors
Nasiri, H., Nasehi, S., & Goudarzi, M.
Affiliations
Sharif University of Technology (IR)
Corresponding author
Nasiri, Hamid
ORCID
Venue
Journal of Big Data
Publisher
Springer Science and Business Media LLC
Published
2019-06-11
Volume
6
Issue
1
ISSN
2196-1115 (ISSN-L)
DOI
10.1186/s40537-019-0215-2
OpenAlex
W2952604789
Cited by
117
References
1
Citations by year
2026: 9 · 2025: 13 · 2024: 9 · 2023: 24 · 2022: 25 · 2021: 21 · 2020: 14 · 2019: 2
FWCI
13.74
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal Of Big Data (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
Iran National Science Foundation (96015834)
Topics
  • Physical Sciences › Computer Science › Information Systems › Advanced Database Systems and Queries
  • Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
  • Physical Sciences › Computer Science › Computer Networks and Communications › IoT and Edge/Fog Computing
Keywords
distributed stream processing, Internet of Things, smart cities, Apache Storm, Spark Streaming, Apache Flink, real-time data processing, throughput and latency, big data analytics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review