CoChat
Sign Up
Cited by 207Advanced Database Systems and QueriesDistributed systems and fault toleranceCloud Computing and Resource Management

The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.

A survey on the evolution of stream processing systems

Open access
Fragkoulis, Marios, Carbone, Paris, Kalavri, Vasiliki, Katsifodimos, Asterios·2024·The VLDB Journal
DOI 10.1007/s00778-023-00819-8
Open this paper in CoChat — free accountOpen PDF

Abstract

Stream processing has been an active research field for more than 20 years, but it is now witnessing its prime time due to recent successful efforts by the research community and numerous worldwide open-source communities. This survey provides a comprehensive overview of fundamental aspects of stream processing systems and their evolution in the functional areas of out-of-order data management, state management, fault tolerance, high availability, load management, elasticity, and reconfiguration. We review noteworthy past research findings, outline the similarities and differences between the first (’00–’10) and second (’11–’23) generation of stream processing systems, and discuss future trends and open problems.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s00778-023-00819-8

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Modern stream processing frameworks are designed to run in a distributed fashion on commodity hardware in order to scale with massive amounts of data.

    “Modern stream processing frameworks are designed to run in a distributed fashion on commodity hardware in order to scale with massive amounts of data (Fragkoulis et al., 2023).”

    From Benchmarking scalability of stream processing frameworks deployed as microservices in the cloud (2024) · cites this paper as Fragkoulis et al., 2023

    The Introduction says stream processors execute in parallel on distributed clusters and are converging toward scale-out processing of massive streams. It does not specify that those clusters use commodity hardware, so that part of the claim is unsupported. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Apache Storm and Apache Heron provide less sophisticated programming models and weaker fault tolerance mechanisms.

    “Storm and Heron provide less sophisticated programming models and weaker fault tolerance mechanisms (Fragkoulis et al., 2023).”

    From Benchmarking scalability of stream processing frameworks deployed as microservices in the cloud (2024) · cites this paper as Fragkoulis et al., 2023

    Section 4.2.1 says systems such as Storm left state management to programmers, and Section 5.5 contrasts the approximate results of earlier systems with modern systems’ exactly-once state semantics under failure. These passages support a qualified comparison for Storm, but the copy does not address Heron or directly compare the programming models and fault-tolerance mechanisms of both named systems. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 155

  1. Abadi, D.J., Ahmad, Y., Balazinska, M., Çetintemel, U., Cherniack, M., Hwang, J., Lindner, W., Maskey, A., Rasin, A., Ryvkina, E., Tatbul, N., Xing, Y., Zdonik, S.B.: The design of the borealis stream processing engine. In Second Biennial Conference on Innovative Data Systems Research, CIDR 2005, Asilomar, CA, USA, pp. 277–289, (2005)
  2. Abadi, D.J., Carney, D., Çetintemel, U., Cherniack, M., Convey, C., Lee, S., Stonebraker, M., Tatbul, N., Zdonik, S.B.: Aurora: a new model and architecture for data stream management. VLDB J. 12(2), 120–139 (2003) · DOI
  3. Akhter, A., Fragkoulis, M., Katsifodimos, A.: Stateful functions as a service in action. Proc. VLDB Endow. 12(12), 1890–1893 (2019) · DOI
  4. Akidau, T., Balikov, A., Bekiroğlu, K., Chernyak, S., Haberman, J., Lax, R., McVeety, S., Mills, D., Nordstrom, P., Whittle, S.: Millwheel: fault-tolerant stream processing at internet scale. Proc. VLDB Endow. 6(11), 1033–1044 (2013) · DOI
  5. Akidau, T., Begoli, E., Chernyak, S., Hueske, F., Knight, K., Knowles, K., Mills, D., Sotolongo, D.: Watermarks in stream processing systems: semantics and comparative analysis of apache flink and google cloud dataflow. Proc. VLDB Endow. 14(12), 3135–3147 (2021) · DOI · Paper page
  6. Akidau, T., Bradshaw, R., Chambers, C., Chernyak, S., Fernández-Moctezuma, R.J., Lax, R., McVeety, S., Mills, D., Perry, F., Schmidt, E. et al.: The dataflow model: a practical approach to balancing correctness, latency, and cost in massive-scale, unbounded, out-of-order data processing. In: VLDB, (2015) · DOI · Paper page
  7. Ali, M., Chandramouli, B., Goldstein, J., Schindlauer, R.: The extensibility framework in Microsoft StreamInsight. In 2011 IEEE 27th International Conference on Data Engineering, pp. 1242–1253. IEEE, (2011) · DOI
  8. Amini, L., Jain, N., Sehgal, A., Silber, J., Verscheure, O.: Adaptive control of extreme-scale stream processing systems. In 26th IEEE International Conference on Distributed Computing Systems (ICDCS’06), pp. 71–71. IEEE, (2006) · DOI
  9. Arasu, A., Babcock, B., Babu, S., Cieslewicz, J., Datar, M., Ito, K., Motwani, R., Srivastava, U., Widom, J.: Stream: the Stanford data stream management system. Data Stream Manage. Process. High-Speed Data Streams, 317–336 (2016) · DOI
  10. Arasu, A., Babcock, B., Babu, S., Datar, M., Ito, K., Nishizawa, I., Rosenstein, J., Widom, J.: STREAM: the stanford stream data manager. In Halevy, A.Y., Ives, Z.G., Doan, A. (eds.) Proceedings of the 2003 ACM SIGMOD International Conference on Management of Data, San Diego, California, USA, June 9-12, 2003, p. 665. ACM, (2003) · DOI
  11. Arasu, A., Babu, S., Widom, J.: The CQL continuous query language: semantic foundations and query execution. VLDB J. 15(2), 121–142 (2006) · DOI
  12. Arasu, A., Widom, J.: Resource sharing in continuous sliding-window aggregates. In Proceedings of the Thirtieth International Conference on Very Large Data Bases, VLDB, pp. 336–347 (2004) · DOI
  13. Armbrust, M., Das, T., Torres, J., Yavuz, B., Zhu, S., Xin, R., Ghodsi, A., Stoica, I., Zaharia, M.: In Proceedings of the 2018 International Conference on Management of Data, SIGMOD Conference 2018, pp. 601–613. ACM, (2018)
  14. Asyabi, E., Wang, Y., Liagouris, J., Kalavri, V., Bestavros, A.: A new benchmark harness for systematic and robust evaluation of streaming state stores. In Proceedings of the Seventeenth European Conference on Computer Systems, EuroSys ’22, pp. 559-574, New York, NY, USA, (2022). Association for Computing Machinery · DOI
  15. Avnur, R., Hellerstein, J. M.: Eddies: Continuously adaptive query processing. In Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data, pp. 261–272. ACM, (2000) · DOI
  16. Babcock, B., Babu, S., Datar, M., Motwani, R.: Chain: operator scheduling for memory minimization in data stream systems. In Proceedings of the 2003 ACM SIGMOD International Conference on Management of Data, pp. 253–264. ACM, (2003) · DOI
  17. Babcock, B., Babu, S., Datar, M., Motwani, R., Widom, J.: Models and issues in data stream systems. In Proceedings of the Twenty-first ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems, pp. 1–16. ACM, (2002) · DOI
  18. Babcock, B., Datar, M., Motwani, R.: Load shedding for aggregation queries over data streams. In Proceedings of the 20th International Conference on Data Engineering, ICDE 2004, pp. 350–361, (2004)
  19. Babu, S., Motwani, R., Munagala, K., Nishizawa, I., Widom, J.: Adaptive ordering of pipelined stream filters. In Proceedings of the ACM SIGMOD International Conference on Management of Data, pp 407–418, (2004) · DOI
  20. Balazinska, M., Balakrishnan, H., Madden, S.R., Stonebraker, M.: Fault-tolerance in the Borealis distributed stream processing system. ACM TODS 33(1), 44 (2008) · DOI
  21. Barga, R.S., Goldstein, J., Ali, M.H., Hong, M.: Consistent streaming through time: a vision for event stream processing. In Third Biennial Conference on Innovative Data Systems Research, CIDR pp. 363–374 (2007)
  22. Begoli, E., Akidau, T., Hueske, F., Hyde, J., Knight, K., Knowles, K.L.: One SQL to rule them all - an efficient and syntactically idiomatic approach to management of streams and tables. In Proceedings of the 2019 International Conference on Management of Data, SIGMOD Conference 2019, pp. 1757–1772. ACM · DOI
  23. Botan, I., Derakhshan, R., Dindar, N., Haas, L.M., Miller, R.J., Tatbul, N.: SECRET: a model for analysis of the execution semantics of stream processing systems. Proc. VLDB Endow. 3(1), 232–243 (2010) · DOI
  24. Burckhardt, S., Gillum, C., Justo, D., Kallas, K., McMahon, C., Meiklejohn, C.S.: Durable functions: semantics for stateful serverless. Proc. ACM Program. Lang. 5(OOPSLA), 1–27 (2021) · DOI
  25. Carbone, P.: Scalable and Reliable Data Stream Processing. PhD thesis, KTH Royal Institute of Technology (2018)
  26. Carbone, P., Ewen, S., Fóra, G., Haridi, S., Richter, S., Tzoumas, K.: State management in Apache Flink®: consistent stateful distributed stream processing. Proc. VLDB Endow. 10(12), 1718–1729 (2017) · DOI · Paper page
  27. Carbone, P., Ewen, S., Haridi, S., Katsifodimos, A., Markl, V., Tzoumas, K.: Apache Flink: Stream and batch processing in a single engine. IEEE Data Eng. Bull., 38 (2015)
  28. Carbone, P., Fóra, G., Ewen, S., Haridi, S., Tzoumas, K.: Lightweight asynchronous snapshots for distributed dataflows. (2015) arXiv preprint arXiv:1506.08603 · arXiv
  29. Carbone, P., Fragkoulis, M., Kalavri, V., Katsifodimos, A.: Beyond analytics: The evolution of stream processing systems. In Proceedings of the 2020 International Conference on Management of Data, SIGMOD Conference 2020, pp. 2651–2658 · DOI
  30. Carney, D., Çetintemel, U., Rasin, A., Zdonik, S. B., Cherniack, M., Stonebraker, M.: Operator scheduling in a data stream manager. In Proceedings of 29th International Conference on Very Large Data Bases, VLDB 2003, pp. 838–849, (2003) · DOI
  31. Çetintemel, U., Abadi, D., Ahmad, Y., Balakrishnan, H., Balazinska, M., Cherniack, M., Hwang, J.-H., Madden, S., Maskey, A., Rasin, A. et al.: The aurora and borealis stream processing engines. In Data Stream Management, pp. 337–359. Springer (2016) · DOI
  32. Çetintemel, U., Du, J., Kraska, T., Madden, S., Maier, D., Meehan, J., Pavlo, A., Stonebraker, M., Sutherland, E., Tatbul, N., Tufte, K., Wang, H., Zdonik, S.B.: S-store: a streaming newsql system for big velocity applications. Proc. VLDB Endow. 7(13), 1633–1636 (2014) · DOI
  33. Chandramouli, B., Goldstein, J.: Shrink—prescribing resiliency solutions for streaming. Proc. VLDB Endow. 10(5), 505–516 (2017) · DOI
  34. Chandramouli, B., Goldstein, J., Barnett, M., DeLine, R., Platt, J.C., Terwilliger, J.F., Wernsing, J.: Trill: a high-performance incremental query processor for diverse analytics. Proc. VLDB Endow. 8(4), 401–412 (2014) · DOI
  35. Chandramouli, B., Goldstein, J., Li, Y.: Impatience is a virtue: revisiting disorder in high-performance log analytics. In 34th IEEE International Conference on Data Engineering, ICDE 2018, pp. 677–688, (2018) · DOI
  36. Chandramouli, B., Goldstein, J., Maier, D.: On-the-fly progress detection in iterative stream queries. Proc. VLDB Endow. 2(1), 241–252 (2009) · DOI
  37. Chandramouli, B., Prasaad, G., Kossmann, D., Levandoski, J.J., Hunter, J., Barnett, M.: FASTER: a concurrent key-value store with in-place updates. In Proceedings of the 2018 International Conference on Management of Data, SIGMOD Conference 2018, pp. 275–290 · DOI
  38. Chandrasekaran, S., Cooper, O., Deshpande, A., Franklin, M.J., Hellerstein, J.M., Hong, W., Krishnamurthy, S., Madden, S., Reiss, F., Shah, M.A.: TelegraphCQ: Continuous dataflow processing. In Proceedings of the 2003 ACM SIGMOD International Conference on Management of Data, p. 668 · DOI
  39. Chandy, K.M., Lamport, L.: Distributed snapshots: determining global states of distributed systems. ACM Trans. Comput. Syst. (TOCS) 3(1), 63–75 (1985) · DOI
  40. Chang, F., Dean, J., Ghemawat, S., Hsieh, W.C., Wallach, D.A., Burrows, M., Chandra, T., Fikes, A., Gruber, R.E.: Bigtable: a distributed storage system for structured data. ACM TOCS 26(2), 26 (2008) · DOI
  41. Chen, J., DeWitt, D.J., Tian, F., Wang, Y.: NiagaraCQ: a scalable continuous query system for internet databases. pp. 379–390 · DOI
  42. Cherniack, M., Balakrishnan, H., Balazinska, M., Carney, D., Çetintemel, U., Xing, Y., Zdonik, S. B.: Scalable distributed stream processing. In First Biennial Conference on Innovative Data Systems Research, CIDR (2003)
  43. Corbett, J.C., Dean, J., Epstein, M., Fikes, A., Frost, C., Furman, J.J., Ghemawat, S., Gubarev, A., Heiser, C., Hochschild, P., et al.: Spanner: Google’s globally distributed database. ACM TOCS 31(3), 22 (2013) · DOI
  44. Cranor, C. D., Johnson, T., Spatscheck, O., Shkapenyuk, V.: Gigascope: A stream database for network applications. In Proceedings of the 2003 ACM SIGMOD International Conference on Management of Data, pp. 647–651 · DOI
  45. Cugola, G., Margara, A.: Processing flows of information: from data stream to complex event processing. ACM Comput. Surv. 44(3), 61 (2012) · DOI
  46. da Silva, G.J., Zheng, F., Debrunner, D., Wu, K., Dogaru, V., Johnson, E., Spicer, M., Sariyüce, A.E.: Consistent regions: guaranteed tuple processing in IBM streams. Proc. VLDB. Endow. 9(13), 1341–1352 (2016) · DOI
  47. Das, A., Gehrke, J., Riedewald, M.: Approximate join processing over data streams. In Proceedings of the 2003 ACM SIGMOD International Conference on Management of Data, pp. 40–51 · DOI
  48. Dayarathna, M., Perera, S.: Recent advancements in event processing. ACM Comput. Surv. 51(2), 35 (2018)
  49. de Assuncao, M.D., da Silva Veith, A., Buyya, R.: Distributed data stream processing and edge computing: a survey on resource elasticity and future directions. J. Netw. Comput. Appl. 103, 1–17 (2018) · DOI
  50. de Heus, M., Psarakis, K., Fragkoulis, M., Katsifodimos, A.: Distributed transactions on serverless stateful functions. In Proceedings of the 15th ACM International Conference on Distributed and Event-based Systems, pp. 31–42 (2021) · DOI
  51. Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. In 6th Symposium on Operating System Design and Implementation (OSDI 2004), pp. 137–150. USENIX Association
  52. Elnozahy, E.N.M., Alvisi, L., Wang, Y.-M., Johnson, D.B.: A survey of rollback-recovery protocols in message-passing systems. ACM Comput. Surv. 34(3), 34 (2002) · DOI
  53. Farhat, O., Daudjee, K., Querzoni, L.: Klink: Progress-aware scheduling for streaming data systems. In Proceedings of the 2021 ACM SIGMOD International Conference on Management of Data, pp. 485–498 · DOI
  54. Fernandez, R.C., Migliavacca, M., Kalyvianaki, E., Pietzuch, P.R.: Integrating scale out and fault tolerance in stream processing using operator state management. In Proceedings of the ACM SIGMOD International Conference on Management of Data, SIGMOD (2013), pp. 725–736
  55. Fernandez, R.C., Migliavacca, M., Kalyvianaki, E., Pietzuch, P.R.: Making state explicit for imperative big data processing. In 2014 USENIX Annual Technical Conference, USENIX ATC ’14, pp. 49–60
  56. Floratou, A., Agrawal, A., Graham, B., Rao, S., Ramasamy, K.: Dhalion: self-regulating stream processing in heron. Proc. VLDB Endow. 10(12), 1825–1836 (2017) · DOI
  57. Fu, T.Z.J., Ding, J., Ma, R.T.B., Winslett, M., Yang, Y., Zhang, Z.: DRS: auto-scaling for real-time stream analytics. IEEE/ACM Trans. Netw. 25(6), 15 (2017) · DOI
  58. Fu, X., Ghaffar, T., Davis, J.C., Lee, D.: Edgewise: a better stream processing engine for the edge. In 2019 USENIX Annual Technical Conference, USENIX ATC 2019, pp. 929–946 (2019)
  59. Garofalakis, M., Gehrke, J., Rastogi, R.: Data stream management: processing high-speed data streams. Springer, Berlin (2007)
  60. Gedik, B.: Partitioning functions for stateful data parallelism in stream processing. VLDBJ 23(4), 517–539 (2014) · DOI
  61. Gedik, B., Schneider, S., Hirzel, M., Wu, K.L.: Elastic scaling for data stream processing. IEEE Trans. Parallel Distrib. Syst. 25(6), 17 (2014) · DOI
  62. Gencer, C., Topolnik, M., Durina, V., Demirci, E., Kahveci, E.B., Gürbüz, A., Bartók, J., Gierlach, G., Hartman, F., Yilmaz, U., Lukás, O., Dogan, M., Mandouh, M., Fragkoulis, M., Katsifodimos, A.: Hazelcast jet Low-latency stream processing at the 99.99th percentile. Proc. VLDB Endow. 14(12), 3110–3121 (2021) · DOI · Paper page
  63. Gray, J., Siewiorek, D.P.: High-availability computer systems. Computer 24(9), 10 (1991) · DOI
  64. Gu, R., Yin, H., Zhong, W., Yuan, C., Huang, Y.: Meces: latency-efficient rescaling via prioritized state migration for stateful distributed stream processing systems. In: USENIX Annual Technical Conference. USENIX ATC, pp. 539–556 (2022)
  65. Gu, Y., Zhang, Z., Ye, F., Yang, H., Kim, M., Lei, H., Liu, Z.: An empirical study of high availability in stream processing systems. In Middleware 2008, ACM/IFIP/USENIX 9th International Middleware Conference, p. 23
  66. Gulisano, V., Jiménez-Peris, R., Patiño-Martínez, M., Soriente, C., Valduriez, P.: StreamCloud: an elastic and scalable data streaming system. IEEE Trans. Parallel Distrib. Syst. 23(12), 15 (2012) · DOI
  67. Hammad, M.A., Franklin, M.J., Aref, W.G., Elmagarmid, A.K.: Scheduling for shared window joins over data streams. In Proceedings of 29th International Conference on Very Large Data Bases, VLDB (2003), pp. 297–308 · DOI
  68. Heinze, T., Aniello, L., Querzoni, L., Jerzak, Z.: Cloud-based data stream processing. In The 8th ACM International Conference on Distributed Event-Based Systems, DEBS ’14, pp. 238–245 · DOI
  69. Heinze, T., Jerzak, Z., Hackenbroich, G., Fetzer, C.: Latency-aware elastic scaling for distributed data stream processing systems. In The 8th ACM International Conference on Distributed Event-Based Systems, DEBS ’14, pp. 13–22
  70. Heinze, T., Pappalardo, V., Jerzak, Z., Fetzer, C.: Auto-scaling techniques for elastic data stream processing. In The 8th ACM International Conference on Distributed Event-Based Systems, DEBS ’14, pp. 318–321
  71. Heinze, T., Zia, M., Krahn, R., Jerzak, Z., Fetzer, C.: An adaptive replication scheme for elastic data stream processing systems. In Proceedings of the 9th ACM International Conference on Distributed Event-Based Systems, DEBS ’15, pp. 150–161 · DOI
  72. Hirzel, M., Baudart, G., Bonifati, A., Della Valle, E., Sakr, S., Akrivi Vlachou, A.: Stream processing languages in the big data era. SIGMOD Record, 47(2), (2018) · DOI
  73. Hoffmann, M., Lattuada, A., McSherry, F., Kalavri, V., Liagouris, J., Roscoe, T.: Megaphone: Latency-conscious state migration for distributed streaming dataflows. Proc. VLDB Endow. 12(9), 1002–1015 (2019) · DOI
  74. Hwang, J., Balazinska, M., Rasin, A., Çetintemel, U., Stonebraker, M., Zdonik, S.B.: High-availability algorithms for distributed stream processing. In Proceedings of the 21st International Conference on Data Engineering, ICDE 2005, pp. 779–790
  75. Hwang, J., Xing, Y., Çetintemel, U., Zdonik, S.B.: A cooperative, self-configuring high-availability solution for stream processing. In Proceedings of the 23rd International Conference on Data Engineering, ICDE 2007, pp. 176–185 · DOI
  76. Johnson, T., Muthukrishnan, S., Shkapenyuk, V., Spatscheck, O.: A heartbeat mechanism and its application in gigascope. In Proceedings of the 31st International Conference on Very Large Data Bases, pp. 1079–1088, (2005)
  77. Kalavri, V., Liagouris, J.: In support of workload-aware streaming state management. In 12th USENIX Workshop on Hot Topics in Storage and File Systems (HotStorage 20), (2020)
  78. Kalavri, V., Liagouris, J., Hoffmann, M., Dimitrova, D.C., Forshaw, M., Roscoe, T.: Three steps is all you need: fast, accurate, automatic scaling decisions for distributed streaming dataflows. In 13th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2018, pp. 783–798
  79. Kang, J., Naughton, J.F., Viglas, S.: Evaluating window joins over unbounded streams. In Proceedings of the 19th International Conference on Data Engineering, pp. 341–352 · DOI
  80. Katsifodimos, A., Fragkoulis, M.: Operational stream processing: towards scalable and consistent event-driven applications. In Advances in Database Technology - 22nd International Conference on Extending Database Technology, EDBT 2019, pp. 682–685
  81. Katsipoulakis, N.R., Labrinidis, A., Chrysanthis, P.K.: Concept-driven load shedding: reducing size and error of voluminous and variable data streams. In IEEE International Conference on Big Data (IEEE BigData 2018), pp. 418–427 · DOI
  82. Khoshkbarforoushha, A., Khosravian, A., Ranjan, R.: Elasticity management of streaming data analytics flows on clouds. J. Comput. Syst. Sci. 89, 24–40 (2017) · DOI
  83. Killalea, T.: The hidden dividends of microservices. ACM Queue, (2016) · DOI
  84. Kleppmann, M., Beresford, A.R., Svingen, B.: Online event processing: achieving consistency where distributed transactions have failed. ACM Queue, (2019) · DOI
  85. Kotselidis, C., Diamantopoulos, S., Mylonas, G.: A big data software paradigm for heterogeneous cloud deployments. Inf. Intell. Syst. Appl. 1(1), 6–10 (2020)
  86. Krishnamurthy, S., Franklin, M.J., Davis, J., Farina, D., Golovko, P., Li, A., Thombre, N.: Continuous analytics over discontinuous streams. In Proceedings of the ACM SIGMOD International Conference on Management of Data, SIGMOD 2010, pp. 1081–1092 · DOI
  87. Kulkarni, S., Bhagat, N., Fu, M., Kedigehalli, V., Kellogg, C., Mittal, S., Patel, J. M., Ramasamy, K., Taneja, S.: Twitter heron: Stream processing at scale. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, pp. 239–250 · DOI · Paper page
  88. Kung, H. T., Blackwell, T., Chapman, A.: Credit-based flow control for ATM networks: Credit update protocol, adaptive credit allocation and statistical multiplexing. In Proceedings of the ACM SIGCOMM 1994 Conference on Communications Architectures, Protocols and Applications, pp. 101–114 · DOI
  89. Kwon, Y., Balazinska, M., Greenberg, A.G.: Fault-tolerant stream processing using a distributed, replicated file system. Proc. VLDB Endow. 1(1), 574–585 (2008) · DOI
  90. Laigner, R., Zhou, Y., Salles, M.A.V., Liu, Y., Kalinowski, M.: Data management in microservices: State of the practice, challenges, and research directions. Proc. VLDB Endow. 14(13), (2021) · DOI
  91. Lamport, L.: Time, clocks, and the ordering of events in a distributed system. Commun. ACM 21(7), 558–565 (1978) · DOI · Paper page
  92. Lee, G., Maeng, J., Park, J., Seo, J., Cho, H., Yang, Y., Um, T., Lee, J., Lee, J.W., Chun, B.-G.: Flowkv: a semantic-aware store for large-scale state management of stream processing engines. In Proceedings of the Eighteenth European Conference on Computer Systems, EuroSys ’23, pp. 768–783, New York, NY, USA, (2023). Association for Computing Machinery · DOI
  93. Li, J., Maier, D., Tufte, K., Papadimos, V., Tucker, P.A.: No pane, no gain: efficient evaluation of sliding-window aggregates over data streams. SIGMOD Rec. 34(1), 39–44 (2005) · DOI
  94. Li, J., Tufte, K., Shkapenyuk, V., Papadimos, V., Johnson, T., Maier, D.: Out-of-order processing: a new architecture for high-performance stream systems. Proc. VLDB Endow. 1(1), 274–288 (2008) · DOI
  95. Lin, W., Fan, H., Qian, Z., Xu, J., Yang, S., Zhou, J., Zhou, L.: Streamscope: continuous reliable distributed processing of big data streams. In 13th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2016, pp. 439–453
  96. Liu, X., Buyya, R.: Resource management and scheduling in distributed stream processing systems: a taxonomy, review, and future directions. ACM Comput. Surv. (CSUR) 53(3), 1–41 (2020) · DOI
  97. Lohrmann, B., Janacik, P., Kao, O.: Elastic stream processing with latency guarantees. In 35th IEEE International Conference on Distributed Computing Systems, ICDCS 2015, pp. 399–410 · DOI
  98. Mai, L., Zeng, K., Potharaju, R., Xu, L., Suh, S., Venkataraman, S., Costa, P., Kim, T., Muthukrishnan, S., Kuppa, V., et al.: Chi: a scalable and programmable control plane for distributed stream processing systems. Proc. VLDB Endow. 11(10), 1303–1316 (2018) · DOI
  99. Maier, D., Li, J., Tucker, P.A., Tufte, K., Papadimos, V.: Semantics of data streams and operators. In Database Theory - ICDT 2005, 10th International Conference, vol. 3363, pp. 37–52 · DOI
  100. Manku, G. S., Motwani, R.: Approximate frequency counts over data streams. In Proceedings of 28th International Conference on Very Large Data Bases, VLDB 2002, pp. 346–357 · DOI
  101. Matteis, T.D., Mencagli, G.: Elastic scaling for distributed latency-sensitive data stream operators. In 25th Euromicro International Conference on Parallel, Distributed and Network-based Processing, PDP 2017, pp. 61–68 · DOI
  102. Meehan, J., Tatbul, N., Zdonik, S., Aslantas, C., Çetintemel, U., Du, J., Kraska, T., Madden, S., Maier, D., Pavlo, A., Stonebraker, M., Tufte, K., Wang, H.: S-Store: streaming meets transaction processing. Proc. VLDB Endow. 8(13), 2134–2145 (2015) · DOI
  103. Mei, Y., Cheng, L., Talwar, V., Levin, M.Y., Jacques-Silva, G., Simha, N., Banerjee, A., Smith, B., Williamson, T., Yilmaz, S., Chen, W., Chen, G.J.: Turbine: Facebook’s service management platform for stream processing. In 36th IEEE International Conference on Data Engineering, ICDE 2020, pp. 1591–1602 · DOI
  104. Migliavacca, M., Eyers, D., Bacon, J., Papagiannis, Y., Shand, B., Pietzuch, P.: SEEP: scalable and elastic event processing. In Middleware Posters and Demos, (2010) · DOI
  105. Monte, B.D., Zeuch, S., Rabl, T., Markl, V.: Rhino: efficient management of very large distributed state for stream processing engines. In Proceedings of the 2020 International Conference on Management of Data, SIGMOD Conference, pp. 2471–2486 · DOI
  106. Murray, D.G., McSherry, F., Isaacs, R., Isard, M., Barham, P., Abadi, M.: Naiad: a timely dataflow system. In ACM SIGOPS 24th Symposium on Operating Systems Principles, SOSP, pp. 439–455 · DOI
  107. Murray, D.G., McSherry, F., Isard, M., Isaacs, R., Barham, P., Abadi, M.: Incremental, iterative data processing with timely dataflow. Commun. ACM 59(10), 75–83 (2016) · DOI
  108. Mutschler, C., Philippsen, M.: Reliable speculative processing of out-of-order event streams in generic publish/subscribe middlewares. In The 7th ACM International Conference on Distributed Event-Based Systems, DEBS ’13, pp. 147–158 · DOI
  109. Neumeyer, L., Robbins, B., Nair, A., Kesari, A.: S4: distributed stream computing platform. In ICDMW 2010, The 10th IEEE International Conference on Data Mining Workshops, pp. 170–177 · DOI
  110. Noghabi, S.A., Paramasivam, K., Pan, Y., Ramesh, N., Bringhurst, J., Gupta, I., Campbell, R.H.: Stateful scalable stream processing at LinkedIn. Proc. VLDB Endow. 10(12), 1634–1645 (2017) · DOI · Paper page
  111. O’Neil, P., Cheng, E., Gawlick, D., O’Neil, E.: The log-structured merge-tree (lsm-tree). Acta Inf. 33(4), 351–385 (1996) · DOI
  112. Palyvos-Giannas, D., Havers, B., Papatriantafilou, M., Gulisano, V.: Ananke: a streaming framework for live forward provenance. Proc. VLDB Endow. 14(3), 391–403 (2020) · DOI
  113. Palyvos-Giannas, D., Mencagli, G., Papatriantafilou, M., Gulisano, V.: Lachesis: a middleware for customizing os scheduling of stream processing queries. In Proceedings of the 22nd International Middleware Conference, pp. 365–378 (2021) · DOI
  114. Psarakis, K., Zorgdrager, W., Fragkoulis, M., Salvaneschi, G., Katsifodimos, A.: Stateful entities: object-oriented cloud applications as distributed dataflows. In 27th International Conference on Extending Database Technology, pp. 15–21, (2024)
  115. Qian, Z., He, Y., Su, C., Wu, Z., Zhu, H., Zhang, T., Zhou, L., Yu, Y., Zhang, Z.: Timestream: reliable stream computation in the cloud. In Eighth Eurosys Conference 2013, EuroSys ’13, pp. 1–14 · DOI
  116. Raman, V., Raman, B., Hellerstein, J. M.: Online dynamic reordering for interactive data processing. In VLDB’99, Proceedings of 25th International Conference on Very Large Data Bases, pp. 709–720, (1999)
  117. Rameshan, N., Liu, Y., Navarro, L., Vlassov, V.: Hubbub-scale: towards reliable elastic scaling under multi-tenancy. In IEEE/ACM 16th International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2016, pp. 233–244 (2016) · DOI
  118. Rivetti, N., Anceaume, E., Busnel, Y., Querzoni, L., Sericola, B.: Online scheduling for shuffle grouping in distributed stream processing systems. In Proceedings of the 17th International Middleware Conference, pp. 1–12, (2016) · DOI
  119. Rivetti, N., Querzoni, L., Anceaume, E., Busnel, Y., Sericola, B.: Efficient key grouping for near-optimal load balancing in stream processing systems. In Proceedings of the 9th ACM International Conference on Distributed Event-Based Systems, pp. 80–91, (2015) · DOI
  120. Röger, H., Mayer, R.: A comprehensive survey on parallelization and elasticity in stream processing. ACM Comput. Surv. 52(2), 1–37 (2019) · DOI
  121. Ryvkina, E., Maskey, A., Cherniack, M., Zdonik, S. B.: Revision processing in a stream processing engine: a high-level design. In Proceedings of the 22nd International Conference on Data Engineering, ICDE 2006, p. 141 · DOI
  122. Shah, M.A., Hellerstein, J.M., Brewer, E.A.: Highly-available, fault-tolerant, parallel dataflows. In Weikum, G., König, A.C. Deßloch, S. (eds.), Proceedings of the ACM SIGMOD International Conference on Management of Data, pp. 827–838 · DOI
  123. Shah, M.A., Hellerstein, J.M., Chandrasekaran, S., Franklin, M.J.: Flux: an adaptive partitioning operator for continuous query systems. In Proceedings of the 19th International Conference on Data Engineering, pp. 25–36
  124. Silvestre, P.F., Fragkoulis, M., Spinellis, D., Katsifodimos, A.: Clonos: consistent causal recovery for highly-available streaming dataflows. In SIGMOD ’21: International Conference on Management of Data, pp. 1637–1650 · DOI
  125. Singh, G., Chelini, L., Corda, S., Javed Awan, A., Stuijk, S., Jordans, R., Corporaal, H., Boonstra, A.-J.: A review of near-memory computing architectures: opportunities and challenges. In 2018 21st Euromicro Conference on Digital System Design (DSD), pp. 608–617, Prague, August (2018). IEEE · DOI
  126. Spenger, J., Carbone, P., Haller, P.: Portals: an extension of dataflow streaming for stateful serverless. In Proceedings of the 2022 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software, pp. 153–171, (2022) · DOI
  127. Spenger, J., Huang, C., Haller, P., Carbone, P.: Portals: a showcase of multi-dataflow stateful serverless. Proc. VLDB Endow. 16(12), 4054–4057 (2023) · DOI
  128. Srivastava, U., Widom, J.: Flexible time management in data stream systems. In Proceedings of the Twenty-third ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems, pp. 263–274 · DOI
  129. Su, L., Zhou, Y.: Tolerating correlated failures in massively parallel stream processing engines. In 32nd IEEE International Conference on Data Engineering, ICDE 2016, pp. 517–528 · DOI
  130. Tatbul, N., Çetintemel, U., Zdonik, S., Cherniack, M., Stonebraker, M.: Load shedding in a data stream manager. In VLDB, (2003) · DOI
  131. Tatbul, N., Çetintemel, U., Zdonik, S. B.: Staying FIT: efficient load shedding techniques for distributed stream processing. In Proceedings of the 33rd International Conference on Very Large Data Bases 2007, pp. 159–170
  132. Tatbul, N., Zdonik, S.B.: Window-aware load shedding for aggregation queries over data streams. In Proceedings of the 32nd International Conference on Very Large Data Bases 2006, pp. 799–810
  133. Tatbul, N., Zdonik, S.B., Meehan, J., Aslantas, C., Stonebraker, M., Tufte, K., Giossi, C., Quach, H.: Handling shared, mutable state in stream processing with correctness guarantees. IEEE Data Eng. Bull. 38, 94–104 (2015)
  134. Terry, D.B., Goldberg, D., Nichols, D.A., Oki, B.M.: Continuous queries over append-only databases. In Proceedings of the 1992 ACM SIGMOD International Conference on Management of Data, pp. 321–330 · DOI
  135. Thomas, J., Hanrahan, P., Zaharia, M.: Fleet: a framework for massively parallel streaming on FPGAs. In ASPLOS ’20: Architectural Support for Programming Languages and Operating Systems, pp. 639–651
  136. To, Q.-C., Soto, J., Markl, V.: A survey of state management in big data processing systems. VLDBJ 27(6), 847–872 (2018) · DOI
  137. Tolosana-Calasanz, R., Montes, J.D., Rana, O.F., Parashar, M.: Feedback-control and queueing theory-based resource management for streaming applications. IEEE Trans. Parallel Distrib. Syst. 28, 1061–1075 (2017) · DOI
  138. Toshniwal, A., Taneja, S., Shukla, A., Ramasamy, K., Patel, J.M., Kulkarni, S., Jackson, J., Gade, K., Fu, M., Donham, J. et al.: Storm @ Twitter. In SIGMOD, (2014) · DOI · Paper page
  139. Traub, J., Grulich, P.M., Cuellar, A.R., Breß, S., Katsifodimos, A., Rabl, T., Markl, V.: Efficient window aggregation with general stream slicing. In Advances in Database Technology - 22nd International Conference on Extending Database Technology, EDBT 2019, pp. 97–108
  140. Tu, Y., Liu, S., Prabhakar, S., Yao, B.: Load shedding in stream databases: a control-based approach. In Proceedings of the 32nd International Conference on Very Large Data Bases 2006, pp. 787–798
  141. Tucker, P.A., Maier, D., Sheard, T., Fegaras, L.: Exploiting punctuation semantics in continuous data streams. IEEE Trans. Knowl. Data Eng. 15(3), 555–568 (2003) · DOI
  142. Urhan, T., Franklin, M.J.: Dynamic pipeline scheduling for improving interactive query performance. In VLDB 2001, Proceedings of 27th International Conference on Very Large Data Bases, pp. 501–510
  143. Urhan, T., Franklin, M.J.: Xjoin: a reactively-scheduled pipelined join operator. IEEE Data Eng. Bull., 23, (2000)
  144. Venkataraman, S., Panda, A., Ousterhout, K., Armbrust, M., Ghodsi, A., Franklin, M.J., Recht, B., Stoica, I.: Drizzle: Fast and adaptable stream processing at scale. In Proceedings of the 26th Symposium on Operating Systems Principles 2017, pp. 374–389 · DOI
  145. Verheijde, J., Karakoidas, V., Fragkoulis, M., Katsifodimos, A.: S-QUERY: opening the black box of internal stream processor state. In 38th IEEE International Conference on Data Engineering, ICDE 2022, pp. 1314–1327 · DOI
  146. Wu, Y., Tan, K.-L.: ChronoStream: Elastic stateful stream computation in the cloud. In ICDE, (2015) · DOI
  147. Xu, L., Peng, B., Gupta, I.: Stela: enabling stream processing systems to scale-in and scale-out on-demand. In 2016 IEEE International Conference on Cloud Engineering, IC2E 2016, pp. 22–31 · DOI
  148. Xu, L., Venkataraman, S., Gupta, I., Mai, L., Potharaju, R.: Move fast and meet deadlines: fine-grained real-time stream processing with cameo. In 18th USENIX Symposium on Networked Systems Design and Implementation (NSDI 21), pp. 389–405, (2021)
  149. Zaharia, M., Das, T., Li, H., Hunter, T., Shenker, S., Stoica, I.: Discretized streams: fault-tolerant streaming computation at scale. In ACM SIGOPS 24th Symposium on Operating Systems Principles, SOSP’13, pp. 423–438
  150. Zaharia, M., Das, T., Li, H., Shenker, S., Stoica, I.: Discretized streams: an efficient and fault-tolerant model for stream processing on large clusters. In USENIX HotCloud, (2012) · DOI
  151. Zhang, F., Yang, L., Zhang, S., He, B., Lu, W., Du, X.: Finestream: Fine-grained window-based stream processing on cpu-gpu integrated architectures. In: USENIX Annual Technical Conference. USENIX ATC, pp. 633–647 (2020)
  152. Zhang, S., He, J., Zhou, A.C., He, B.: Briskstream: scaling data stream processing on shared-memory multicore architectures. In Proceedings of the 2019 International Conference on Management of Data, SIGMOD Conference 2019, pp. 705–722 · DOI
  153. Zhang, S., Zhang, F., Wu, Y., He, B., Johns, P.: Hardware-conscious stream processing: a survey. SIGMOD Rec. 48(4), 18–29 (2019) · DOI
  154. Zhang, Z., Gu, Y., Ye, F., Yang, H., Kim, M., Lei, H., Liu, Z.: A hybrid approach to high availability in stream processing systems. In 2010 International Conference on Distributed Computing Systems, ICDCS 2010, pp. 138–148 · DOI
  155. Zhu, X., Serafini, M., Ma, X., Aboulnaga, A., Chen, W., Feng, G.: LiveGraph: a transactional graph storage system with purely sequential adjacency list scans. Proc. VLDB Endow. 13(7), 1020–1034 (2020) · DOI

Cited by

Sorted by citations

Metadata

Authors
Fragkoulis, M., Carbone, P., Kalavri, V., & Katsifodimos, A.
Affiliations
  • RISE Research Institutes of Sweden (SE)
  • Boston University (US)
  • Delft University of Technology (NL)
Corresponding author
Katsifodimos, Asterios
ORCID
Venue
The VLDB Journal
Publisher
Springer Science and Business Media LLC
Published
2023-11-22
Volume
33
Issue
2
Pages
507-541
ISSN
  • 0949-877X (ISSN-L)
  • 1066-8888
DOI
10.1007/s00778-023-00819-8
OpenAlex
W3048826825
Cited by
207
References
1
Citations by year
2026: 74 · 2025: 66 · 2024: 26 · 2023: 19
FWCI
34.01
Type
journal article
Language
en
Open access
Licence
Locations
  • The VLDB Journal (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • Diva portal (Dalarna University Library) (submittedVersion, cc-by)
  • Research Repository (Delft University of Technology) (submittedVersion, cc-by)
  • arXiv (Cornell University)
Funders
  • Nederlandse Organisatie voor Wetenschappelijk Onderzoek (19708)
  • Defense Acquisition Program Administration
Topics
  • Physical Sciences › Computer Science › Information Systems › Advanced Database Systems and Queries
  • Physical Sciences › Computer Science › Computer Networks and Communications › Distributed systems and fault tolerance
  • Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
Keywords
stream processing systems, state management, fault tolerance, high availability, load management, elasticity, reconfiguration, system evolution

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review