The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
Discretized streams: fault-tolerant streaming computation at scale
Open accessAbstract
Many "big data" applications must act on data in real time. Running these applications at ever-larger scales requires parallel platforms that automatically handle faults and stragglers. Unfortunately, current distributed stream processing models provide fault recovery in an expensive manner, requiring hot replication or long recovery times, and do not handle stragglers. We propose a new processing model, discretized streams (D-Streams), that overcomes these challenges. D-Streams enable a parallel recovery mechanism that improves efficiency over traditional replication and backup schemes, and tolerates stragglers. We show that they support a rich set of operators while attaining high per-node throughput similar to single-node systems, linear scaling to 100 nodes, sub-second latency, and sub-second fault recovery. Finally, D-Streams can easily be composed with batch and interactive query models like MapReduce, enabling rich applications that combine these modes. We implement D-Streams in a system called Spark Streaming.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
MapReduce: simplified data processing on large clusters
Dean, J., & Ghemawat, S. (2008) · Communications of the ACM · doi:10.1145/1327452.1327492 · Cited by 18,476
Ghemawat, S., Gobioff, H., & Leung, S.-T. (2003) · ACM SIGOPS Operating Systems Review · doi:10.1145/1165389.945450 · Cited by 4,984
Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J. H., McCauley, M., Franklin, M. J. M., Shenker, S. J., & Stoica, I. G. (2012) · Cited by 3,622
Self-citationDryad: distributed data-parallel programs from sequential building blocks
Isard, M., Budiu, M., Yu, Y., Birrell, A., & Fetterly, D. (2007) · Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007 · doi:10.1145/1272996.1273005 · Cited by 2,432
Ghemawat, S., Gobioff, H., & Leung, S.-T. (2003) · Proceedings of the nineteenth ACM symposium on Operating systems principles · doi:10.1145/945445.945450 · Cited by 1,423
TelegraphCQ: Continuous Dataflow Processing for an Uncertain World.
Chandrasekaran, S., Cooper, O., Deshpande, A., Franklin, M. J. M., Hellerstein, J. M., Hong, W., Krishnamurthy, S., Madden, S. R., Raman, V., Reiss, F. R., et al. (2003) · Conference on Innovative Data Systems Research · Cited by 947
S4: Distributed Stream Computing Platform
Neumeyer, L., Robbins, B., Nair, A., & Kesari, A. (2010) · 2010 IEEE International Conference on Data Mining Workshops · doi:10.1109/icdmw.2010.172 · Cited by 900
Monitoring Streams — A New Class of Data Management Applications
Carney, D., Çetintemel, U., Cherniack, M., Convey, C., Lee, S.-D., Seidman, G., Tatbul, N., Zdonik, S., & Stonebraker, M. R. (2002) · Elsevier eBooks · doi:10.1016/b978-155860869-6/50027-5 · Cited by 842
Yu, Y., Isard, M., Fetterly, D., Budiu, M., Erlingsson, Ú., Gunda, P. K., & Currey, J. (2008) · doi:10.5555/1855741.1855742 · Cited by 689
Condie, T., Conway, N., Alvaro, P., Hellerstein, J. M., Elmeleegy, K., & Sears, R. (2010) · doi:10.5555/1855711.1855732 · Cited by 673
Cited by
Apache Spark: A unified engine for big data processing
Zaharia, M., Xin, R. S., Wendell, P., Das, T., Armbrust, M., Dave, A., Meng, X., Rosen, J., Venkataraman, S., Franklin, M. J., et al. (2016) · Communications of the ACM · doi:10.1145/2934664 · Cited by 2,757
Self-citationAkidau, T., Bradshaw, R., Chambers, C., Chernyak, S., Fernández-Moctezuma, R. J., Lax, R., McVeety, S., Mills, D., Perry, F., Schmidt, E., et al. (2015) · Proceedings of the VLDB Endowment · doi:10.14778/2824032.2824076 · Cited by 750
Naiad: a timely dataflow system
Murray, D. G., McSherry, F., Isaacs, R., Isard, M., Barham, P., & Abadi, M. (2013) · Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles · doi:10.1145/2517349.2522738 · Cited by 656
A Survey on Ensemble Learning for Data Stream Classification
Gomes, H. M., Barddal, J. P., Enembreck, F., & Bifet, A. (2018) · ACM Computing Surveys · doi:10.1145/3054925 · Cited by 579
Resource Management in Fog/Edge Computing: A Survey on Architectures, Infrastructure, and Algorithms
Hong, C.-H., & Varghese, B. (2020) · ACM Computing Surveys · doi:10.1145/3326066 · Cited by 461
Metadata
- Authors
- Zaharia, M., Das, T., Li, H., Hunter, T., Shenker, S., & Stoica, I.
- Venue
- Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles
- Publisher
- ACM
- Published
- 2013-11-03
- Pages
- 423-438
- OpenAlex
- W1976821017
- Cited by
- 982
- References
- 31
- FWCI
- 70.54
- Type
- conference paper
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Discretized streams: fault-tolerant streaming computation at scale. https://app.cochat.ai/papers/10.1145/2517349.2522737
- The paper
- Zaharia, M., Das, T., Li, H., Hunter, T., Shenker, S., & Stoica, I. (2013). Discretized streams: fault-tolerant streaming computation at scale. Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles. https://doi.org/10.1145/2517349.2522737
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
