The DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Integrating scale out and fault tolerance in stream processing using operator state management
Abstract
As users of "big data" applications expect fresh results, we witness a new breed of stream processing systems (SPS) that are designed to scale to large numbers of cloud-hosted machines. Such systems face new challenges: (i) to benefit from the "pay-as-you-go" model of cloud computing, they must scale out on demand, acquiring additional virtual machines (VMs) and parallelising operators when the workload increases; (ii) failures are common with deployments on hundreds of VMs-systems must be fault-tolerant with fast recovery times, yet low per-machine overheads. An open question is how to achieve these two goals when stream queries include stateful operators, which must be scaled out and recovered without affecting query results.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The cost of a cloud: research problems in data center networks
Greenberg, A., Hamilton, J., Maltz, D. A., & Patel, P. (2008) · ACM SIGCOMM Computer Communication Review · doi:10.1145/1496091.1496103 · Cited by 1,621
The design of the borealis stream processing engine
Abadi, D. J., Ahmad, Y., Bałazińska, M., Cherniack, M., Hwang, J.-H., Lindner, W., Maskey, A. S., Rasin, A., Ryvkina, E., Tatbul, N., et al. (2005) · Cited by 1,255
S4: Distributed Stream Computing Platform
Neumeyer, L., Robbins, B., Nair, A., & Kesari, A. (2010) · 2010 IEEE International Conference on Data Mining Workshops · doi:10.1109/icdmw.2010.172 · Cited by 900
Understanding network failures in data centers: measurement, analysis, and implications
Gill, P., Jain, N., & Nagappan, N. (2011) · Proceedings of the ACM SIGCOMM 2011 conference · doi:10.1145/2018436.2018477 · Cited by 714
Big data and cloud computing: current state and future opportunities
Agrawal, D., Das, S., & El Abbadi, A. (2011) · Proceedings of the 14th International Conference on Extending Database Technology · doi:10.1145/1951365.1951432 · Cited by 552
Discretized streams: an efficient and fault-tolerant model for stream processing on large clusters
Zaharia, M., Das, T., Li, H., Shenker, S. J., & Stoica, I. G. (2012) · USENIX conference on Hot Topics in Cloud Ccomputing · Cited by 522
A comparison of join algorithms for log processing in MaPreduce
Blanas, S., Patel, J. M., Ercegovac, V., Rao, J., Shekita, E. J., & Tian, Y. (2010) · Proceedings of the 2010 ACM SIGMOD International Conference on Management of data · doi:10.1145/1807167.1807273 · Cited by 414
Query Processing, Resource Management, and Approximation ina Data Stream Management System
Motwani, R., Widom, J., Arasu, A., Babcock, B., Babu, S., Datar, M., Manku, G. S., Olston, C., Rosenstein, J., & Varma, R. (2002) · Cited by 342
StreamCloud: An Elastic and Scalable Data Streaming System
Gulisano, V., Jimenez-Peris, R., Patino-Martinez, M., Soriente, C., & Valduriez, P. (2012) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2012.24 · Cited by 324
Arasu, A., Maskey, A. S., Cherniack, M., Ryvkina, E., Galvez, E., Stonebraker, M., Maier, D., & Tibbetts, R. (2004) · Proceedings 2004 VLDB Conference · doi:10.1016/b978-012088469-8.50044-9 · Cited by 322
High-Availability Algorithms for Distributed Stream Processing
Hwang, J.-H., Balazinska, M., Rasin, A., Cetintemel, U., Stonebraker, M., & Zdonik, S. (2005) · 21st International Conference on Data Engineering (ICDE'05) · doi:10.1109/icde.2005.72 · Cited by 261
SPC: a distributed, scalable platform for data mining
Amini, L. D., Andrade, H. C. M., Bhagwan, R., Eskesen, F., King, R. P., Selo, P., Park, Y., & Venkatramani, C. (2006) · doi:10.1145/1289612.1289615 · Cited by 158
Elastic scaling of data parallel operators in stream processing
Schneider, S., Andrade, H. C. M., Gedik, B., Biem, A., & Wu, K. (2009) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps.2009.5161036 · Cited by 126
A Cooperative, Self-Configuring High-Availability Solution for Stream Processing
Hwang, J.-H., Xing, Y., Çetintemel, U., & Zdonik, S. (2007) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2007.367863 · Cited by 114
Esc: Towards an Elastic Stream Computing Platform for the Cloud
Satzger, B., Hummer, W., Leitner, P., & Dustdar, S. (2011) · IEEE ... International Conference on Cloud Computing · doi:10.1109/cloud.2011.27 · Cited by 66
Scalable and Low-Latency Data Processing with Stream MapReduce
Brito, A., Martin, A., Knauth, T., Creutz, S., de Brum, D. B., Weigert, S., & Fetzer, C. W. (2011) · IEEE International Conference on Cloud Computing Technology and Science · doi:10.1109/cloudcom.2011.17 · Cited by 63
An empirical study of high availability in stream processing systems
Gu, Y., Zhang, Z., Ye, F., Yang, H., Kim, M., Lei, H., & Liu, Z. (2009) · ACM/IFIP/USENIX international conference on Middleware · Cited by 58
A Hybrid Approach to High Availability in Stream Processing Systems
Zhang, Z., Gu, Y., Ye, F., Yang, H., Kim, M., Lei, H., & Liu, Z. (2010) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs.2010.81 · Cited by 57
Run-time operator state spilling for memory intensive long-running queries
Liu, B., Zhu, Y., & Rundensteiner, E. A. (2006) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/1142473.1142513 · Cited by 47
CEC: Continuous eventual checkpointing for data stream processing operators
Sebepou, Z., & Magoutis, K. (2011) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn.2011.5958214 · Cited by 45
Active Replication at (Almost) No Cost
Martin, A., Fetzer, C. W., & Brito, A. (2011) · IEEE Symposium on Reliable Distributed Systems (SRDS) · doi:10.1109/srds.2011.12 · Cited by 41
Virtualizing Stream Processing
Duller, M., Rellermeyer, J. S., Alonso, G., & Tatbul, N. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-25821-3_14 · Cited by 17
Distributed middleware reliability and fault tolerance support in system S
Wagle, R., Andrade, H. C. M., Hildrum, K. W., Venkatramani, C., & Spicer, M. (2011) · ACM International Conference on Distributed and Event-Based Systems (DEBS) · doi:10.1145/2002259.2002304 · Cited by 16
Cited by
Discretized streams: fault-tolerant streaming computation at scale
Zaharia, M., Das, T., Li, H., Hunter, T., Shenker, S., & Stoica, I. (2013) · Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles · doi:10.1145/2517349.2522737 · Cited by 982
Dias de Assunção, M., da Silva Veith, A., & Buyya, R. (2018) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2017.12.001 · Cited by 304
State management in Apache Flink®: Consistent stateful distributed stream processing
Carbone, P., Ewen, S., Fóra, G., Haridi, S., Richter, S., & Tzoumas, K. (2017) · Proceedings of the VLDB Endowment · doi:10.14778/3137765.3137777 · Cited by 285
Samza: Stateful scalable stream processing at LinkedIn
Noghabi, S. A., Paramasivam, K., Pan, Y., Ramesh, N., Bringhurst, J., Gupta, I., & Campbell, R. H. (2017) · Proceedings of the VLDB Endowment · doi:10.14778/3137765.3137770 · Cited by 276
Consistency and Completeness: Rethinking Distributed Stream Processing in Apache Kafka
Wang, G., Chen, L., Dikshit, A., Gustafson, J., Chen, B., Sax, M. J., Roesler, J., Blee-Goldman, S., Cadonna, B., Mehta, A., et al. (2021) · Proceedings of the 2021 International Conference on Management of Data · doi:10.1145/3448016.3457556 · Cited by 74
Metadata
- Authors
- Castro Fernandez, R., Migliavacca, M., Kalyvianaki, E., & Pietzuch, P.
- Venue
- Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data
- Publisher
- ACM
- Published
- 2013-06-22
- Pages
- 725-736
- ISSN
- 0730-8078 (ISSN-L)
- OpenAlex
- W2152558130
- Cited by
- 338
- References
- 23
- FWCI
- 28.22
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.