The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
Discretized streams: fault-tolerant streaming computation at scale
Open accessAbstract
Many "big data" applications must act on data in real time. Running these applications at ever-larger scales requires parallel platforms that automatically handle faults and stragglers. Unfortunately, current distributed stream processing models provide fault recovery in an expensive manner, requiring hot replication or long recovery times, and do not handle stragglers. We propose a new processing model, discretized streams (D-Streams), that overcomes these challenges. D-Streams enable a parallel recovery mechanism that improves efficiency over traditional replication and backup schemes, and tolerates stragglers. We show that they support a rich set of operators while attaining high per-node throughput similar to single-node systems, linear scaling to 100 nodes, sub-second latency, and sub-second fault recovery. Finally, D-Streams can easily be composed with batch and interactive query models like MapReduce, enabling rich applications that combine these modes. We implement D-Streams in a system called Spark Streaming.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
MapReduce: simplified data processing on large clusters
Dean, J., & Ghemawat, S. (2008) · Communications of the ACM · doi:10.1145/1327452.1327492 · Cited by 18,476
Ghemawat, S., Gobioff, H., & Leung, S.-T. (2003) · ACM SIGOPS Operating Systems Review · doi:10.1145/1165389.945450 · Cited by 4,984
Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J. H., McCauley, M., Franklin, M. J. M., Shenker, S. J., & Stoica, I. G. (2012) · UC Berkeley · Cited by 4,891
Self-citationDryad: distributed data-parallel programs from sequential building blocks
Isard, M., Budiu, M., Yu, Y., Birrell, A., & Fetterly, D. (2007) · Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007 · doi:10.1145/1272996.1273005 · Cited by 2,432
Ghemawat, S., Gobioff, H., & Leung, S.-T. (2003) · Proceedings of the nineteenth ACM symposium on Operating systems principles · doi:10.1145/945445.945450 · Cited by 1,423
TelegraphCQ: Continuous Dataflow Processing for an Uncertain World.
Chandrasekaran, S., Cooper, O., Deshpande, A., Franklin, M. J. M., Hellerstein, J. M., Hong, W., Krishnamurthy, S., Madden, S. R., Raman, V., Reiss, F. R., et al. (2003) · Conference on Innovative Data Systems Research · Cited by 947
S4: Distributed Stream Computing Platform
Neumeyer, L., Robbins, B., Nair, A., & Kesari, A. (2010) · 2010 IEEE International Conference on Data Mining Workshops · doi:10.1109/icdmw.2010.172 · Cited by 900
Monitoring Streams — A New Class of Data Management Applications
Carney, D., Çetintemel, U., Cherniack, M., Convey, C., Lee, S.-D., Seidman, G., Tatbul, N., Zdonik, S., & Stonebraker, M. R. (2002) · Elsevier eBooks · doi:10.1016/b978-155860869-6/50027-5 · Cited by 842
Yu, Y., Isard, M., Fetterly, D., Budiu, M., Erlingsson, Ú., Gunda, P. K., & Currey, J. (2008) · doi:10.5555/1855741.1855742 · Cited by 689
Condie, T., Conway, N., Alvaro, P., Hellerstein, J. M., Elmeleegy, K., & Sears, R. (2010) · doi:10.5555/1855711.1855732 · Cited by 673
Naiad: a timely dataflow system
Murray, D. G., McSherry, F., Isaacs, R., Isard, M., Barham, P., & Abadi, M. (2013) · Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles · doi:10.1145/2517349.2522738 · Cited by 656
MillWheel: fault-tolerant stream processing at internet scale
Akidau, T., Balikov, A., Bekiroğlu, K., Chernyak, S., Haberman, J., Lax, R., McVeety, S., Mills, D., Nordstrom, P., & Whittle, S. (2013) · Proceedings of the VLDB Endowment · doi:10.14778/2536222.2536229 · Cited by 536
Scalable Distributed Stream Processing
Cherniack, M., Balakrishnan, H., Bałazińska, M., Carney, D., Xing, Y., & Zdonik, S. (2003) · Cited by 496
Design and Evaluation of a Real-Time URL Spam Filtering Service
Thomas, K., Grier, C., Ma, J., Paxson, V., & Song, D. (2011) · 2011 IEEE Symposium on Security and Privacy · doi:10.1109/sp.2011.25 · Cited by 468
Large-scale incremental processing using distributed transactions and notifications
Peng, D. N., & Dabek, F. (2010) · doi:10.5555/1924943.1924961 · Cited by 402
Fast crash recovery in RAMCloud
Ongaro, D., Rumble, S. M., Stutsman, R., Ousterhout, J., & Rosenblum, M. (2011) · Proceedings of the Twenty-Third ACM Symposium on Operating Systems Principles · doi:10.1145/2043556.2043560 · Cited by 360
Integrating scale out and fault tolerance in stream processing using operator state management
Castro Fernandez, R., Migliavacca, M., Kalyvianaki, E., & Pietzuch, P. (2013) · Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data · doi:10.1145/2463676.2465282 · Cited by 338
Incoop: MapReduce for incremental computations
Bhatotia, P., Wieder, A., Rodrigues, R., Acar, U. A., & Pasquin, R. (2011) · Proceedings of the 2nd ACM Symposium on Cloud Computing · doi:10.1145/2038916.2038923 · Cited by 298
Flexible time management in data stream systems
Srivastava, U., & Widom, J. (2004) · Proceedings of the twenty-third ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems · doi:10.1145/1055558.1055596 · Cited by 275
STREAM: the stanford stream data manager (demonstration description)
Arasu, A., Babcock, B., Babu, S., Datar, M., Ito, K., Nishizawa, I., Rosenstein, J., & Widom, J. (2003) · Proceedings of the 2003 ACM SIGMOD international conference on Management of data · doi:10.1145/872757.872854 · Cited by 264
High-Availability Algorithms for Distributed Stream Processing
Hwang, J.-H., Balazinska, M., Rasin, A., Cetintemel, U., Stonebraker, M., & Zdonik, S. (2005) · 21st International Conference on Data Engineering (ICDE'05) · doi:10.1109/icde.2005.72 · Cited by 261
TimeStream: reliable stream computation in the cloud
Qian, Z., He, Y., Su, C., Wu, Z., Zhu, H., Zhang, T., Zhou, L., Yu, Y., & Zhang, Z. (2013) · Proceedings of the 8th ACM European Conference on Computer Systems · doi:10.1145/2465351.2465353 · Cited by 227
Highly available, fault-tolerant, parallel dataflows
Shah, M. A., Hellerstein, J. M., & Brewer, E. (2004) · Proceedings of the 2004 ACM SIGMOD international conference on Management of data · doi:10.1145/1007568.1007662 · Cited by 204
Stateful bulk processing for incremental analytics
Logothetis, D., Olston, C., Reed, B. C., Webb, K. C., & Yocum, K. (2010) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/1807128.1807138 · Cited by 172
Fault-tolerance in the borealis distributed stream processing system
Balazinska, M., Balakrishnan, H., Madden, S. R., & Stonebraker, M. (2008) · ACM Transactions on Database Systems · doi:10.1145/1331904.1331907 · Cited by 165
Comet: batched stream processing for data intensive distributed computing
He, B., Yang, M., Guo, Z., Chen, R., Su, B., Lin, W., & Zhou, L. (2010) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/1807128.1807139 · Cited by 161
A Cooperative, Self-Configuring High-Availability Solution for Stream Processing
Hwang, J.-H., Xing, Y., Çetintemel, U., & Zdonik, S. (2007) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2007.367863 · Cited by 114
Microsoft CEP server and online behavioral targeting
Ali, M. H., Gerea, C., Raman, B. S., Sezgin, B., Tarnavski, T., Verona, T., Wang, P., Zabback, P., Ananthanarayan, A., Kirilov, A., et al. (2009) · Proceedings of the VLDB Endowment · doi:10.14778/1687553.1687590 · Cited by 105
Continuous analytics over discontinuous streams
Krishnamurthy, S., Franklin, M. J., Davis, J., Farina, D., Golovko, P., Li, A., & Thombre, N. (2010) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/1807167.1807290 · Cited by 102
Scaling the mobile millennium system in the cloud
Hunter, T., Moldovan, T. M., Zaharia, M., Merzgui, S., Ma, J., Franklin, M. J. M., Abbeel, P., & Bayen, A. M. (2011) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/2038916.2038944 · Cited by 72
Self-citationMeteor Shower: A Reliable Stream Processing System for Commodity Data Centers
Wang, H., Peh, L.-S., Koukoumidis, E., Tao, S., & Chan, M. C. (2012) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps.2012.108 · Cited by 34
Cited by
Apache Spark: A unified engine for big data processing
Zaharia, M., Xin, R. S., Wendell, P., Das, T., Armbrust, M., Dave, A., Meng, X., Rosen, J., Venkataraman, S., Franklin, M. J., et al. (2016) · Communications of the ACM · doi:10.1145/2934664 · Cited by 2,757
Self-citationMLlib: Machine Learning in Apache Spark
Meng, X., Bradley, J. K., Yavuz, B., Sparks, E., Venkataraman, S., Liu, D., Freeman, J., Tsai, D., Amde, M., Owen, S., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.06807 · Cited by 940
Self-citationAkidau, T., Bradshaw, R., Chambers, C., Chernyak, S., Fernández-Moctezuma, R. J., Lax, R., McVeety, S., Mills, D., Perry, F., Schmidt, E., et al. (2015) · Proceedings of the VLDB Endowment · doi:10.14778/2824032.2824076 · Cited by 750
“From the scale enabled by MapReduce [16] and its successors (e.g Hadoop [4], Pig [18], Hive [29], Spark [33]), to the vast body of work on streaming within the SQL community (e.g. query systems [1, 14, 15], windowing [22], data streams [24], time domains [28], semantic models [9]), to the more recent forays in low-lat…”
BackgroundInfluentialNaiad: a timely dataflow system
Murray, D. G., McSherry, F., Isaacs, R., Isard, M., Barham, P., & Abadi, M. (2013) · Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles · doi:10.1145/2517349.2522738 · Cited by 656
Big data analytics for intelligent manufacturing systems: A review
Wang, J., Xu, C., Zhang, J., & Zhong, R. (2022) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2021.03.005 · Cited by 603
Metadata
- Authors
- Zaharia, M., Das, T., Li, H., Hunter, T., Shenker, S., & Stoica, I.
- Venue
- Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles
- Publisher
- ACM
- Published
- 2013-11-03
- Pages
- 423-438
- OpenAlex
- W1976821017
- Cited by
- 982
- References
- 31
- FWCI
- 70.54
- Type
- conference paper
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.