TimeStream: reliable stream computation in the cloud
The DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
TimeStream is a distributed system designed specifically for low-latency continuous processing of big streaming data on a large cluster of commodity machines. The unique characteristics of this emerging application domain have led to a significantly different design from the popular MapReduce-style batch data processing. In particular, we advocate a powerful new abstraction called resilient substitution that caters to the specific needs in this new computation model to handle failure recovery and dynamic reconfiguration in response to load changes. Several real-world applications running on our prototype have been shown to scale robustly with low latency while at the same time maintaining the simple and concise declarative programming model. TimeStream handles an on-line advertising aggregation pipeline at a rate of 700,000 URLs per second with a 2-second delay, while performing sentiment analysis of Twitter data at a peak rate close to 10,000 tweets per second, with approximately 2-second delay.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
MapReduce: simplified data processing on large clusters
Dean, J., & Ghemawat, S. (2008) · Communications of the ACM · doi:10.1145/1327452.1327492 · Cited by 18,476
ZooKeeper: wait-free coordination for internet-scale systems
Hunt, P. G., Konar, M., Junqueira, F., & Reed, B. C. (2010) · USENIX Annual Technical Conference · Cited by 1,968
Pig latin: a not-so-foreign language for data processing
Olston, C., Reed, B., Srivastava, U., Kumar, R., & Tomkins, A. (2008) · Proceedings of the 2008 ACM SIGMOD international conference on Management of data · doi:10.1145/1376616.1376726 · Cited by 1,730
Hive: a warehousing solution over a map-reduce framework
Thusoo, A., Sarma, J. S., Jain, N., Shao, Z., Chakka, P., Anthony, S., Liu, H., Wyckoff, P., & Murthy, R. (2009) · Proceedings of the VLDB Endowment · doi:10.14778/1687553.1687609 · Cited by 1,556
S4: Distributed Stream Computing Platform
Neumeyer, L., Robbins, B., Nair, A., & Kesari, A. (2010) · 2010 IEEE International Conference on Data Mining Workshops · doi:10.1109/icdmw.2010.172 · Cited by 900 · cited 1×
“In contrast to many other systems that deal with off-line batch processing, TimeStream focuses on large-scale lowlatency stream computation, as with a number of recent systems including Storm [2], S4 [20], D-Streams [28], Puma [19] and Flume [14].”
BackgroundSCOPE: easy and efficient parallel processing of massive data sets
Chaiken, R., Jenkins, B., Larson, P.-Å., Ramsey, B., Shakib, D., Weaver, S., & Zhou, J. (2008) · Proceedings of the VLDB Endowment · doi:10.14778/1454159.1454166 · Cited by 727
Yu, Y., Isard, M., Fetterly, D., Budiu, M., Erlingsson, Ú., Gunda, P. K., & Currey, J. (2008) · doi:10.5555/1855741.1855742 · Cited by 689
Self-citationDiscretized streams: an efficient and fault-tolerant model for stream processing on large clusters
Zaharia, M., Das, T., Li, H., Shenker, S. J., & Stoica, I. G. (2012) · USENIX conference on Hot Topics in Cloud Ccomputing · Cited by 522
LINQ: reconciling object, relations and XML in the .NET framework
Meijer, E., Beckman, B., & Bierman, G. M. (2006) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/1142473.1142552 · Cited by 381
FlumeJava: easy, efficient data-parallel pipelines
Chambers, C., Raniwala, A., Perry, F., Adams, S., Henry, R. R., Bradshaw, R., & Weizenbaum, N. (2010) · Proceedings of the 31st ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/1806596.1806638 · Cited by 331 · cited 5×
“[14] CH AMB E R S , C., RA NI WA L A , A., PE R RY, F., ADA M S, S., HE N RY, R. R., BR A DSHAW, R., A N D WE I ZE NBAUM , N. FlumeJava: Easy, ef.cient data-parallel pipelines.”
BackgroundInfluentialIncoop: MapReduce for incremental computations
Bhatotia, P., Wieder, A., Rodrigues, R., Acar, U. A., & Pasquin, R. (2011) · Proceedings of the 2nd ACM Symposium on Cloud Computing · doi:10.1145/2038916.2038923 · Cited by 298 · cited 3×
“[11] BH ATOT I A , P., WIEDE R , A., RO D RI G U E S , R., ACA R , U. A., A N D PASQUI N , R. Incoop: MapReduce for incremental computations.”
MethodInfluentialHigh-Availability Algorithms for Distributed Stream Processing
Hwang, J.-H., Balazinska, M., Rasin, A., Cetintemel, U., Stonebraker, M., & Zdonik, S. (2005) · 21st International Conference on Data Engineering (ICDE'05) · doi:10.1109/icde.2005.72 · Cited by 261
Dynamic Load Distribution in the Borealis Stream Processor
Xing, Y., Zdonik, S., & Hwang, J.-H. (2005) · 21st International Conference on Data Engineering (ICDE'05) · doi:10.1109/icde.2005.53 · Cited by 228 · cited 2×
“Streaming database systems tend to use replication for reliability [7, 8, 17, 24, 26]; such a solution is often costly and relies on strong assumptions on how many failures can occur concurrently.”
BackgroundConsistent Streaming Through Time: A Vision for Event Stream Processing
Barga, R., Goldstein, J., Ali, M., & Hong, M. (2006) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0612115 · Cited by 221 · cited 2×
“3.1 StreamInsight StreamInsight is a powerful platform for developing complex event processing (CEP) applications, which is a Data Stream Management System (DSMS) based on the CEDR [9] research project.”
BackgroundInfluentialHighly available, fault-tolerant, parallel dataflows
Shah, M. A., Hellerstein, J. M., & Brewer, E. (2004) · Proceedings of the 2004 ACM SIGMOD international conference on Management of data · doi:10.1145/1007568.1007662 · Cited by 204 · cited 4×
“The over-provisioning is required to mask failures, and is much cheaper than a hot-standby solution [24] that would require 100% over-provisioning.”
BackgroundInfluentialNectar: automatic management of data and computation in datacenters
Gunda, P. K., Ravindranath, L., Thekkath, R. A., Yu, Y., & Li, Z. (2010) · Cited by 202
Self-citationIBM infosphere streams for scalable, real-time, intelligent transportation services
Biem, A., Bouillet, E., Feng, H., Ranganathan, A., Riabov, A., Verscheure, O., Koutsopoulos, H., & Moran, C. (2010) · Proceedings of the 2010 ACM SIGMOD International Conference on Management of data · doi:10.1145/1807167.1807291 · Cited by 193
Microsoft CEP server and online behavioral targeting
Ali, M. H., Gerea, C., Raman, B. S., Sezgin, B., Tarnavski, T., Verona, T., Wang, P., Zabback, P., Ananthanarayan, A., Kirilov, A., et al. (2009) · Proceedings of the VLDB Endowment · doi:10.14778/1687553.1687590 · Cited by 105 · cited 7×
“Storm does not enforce a deterministic order on how a computation consumes input entries, making it challenging to support strong semantics transparently at the system level; often programmers have to implement a set of APIs in order to support strong semantics.”
BackgroundMethodInfluentialDryadInc: reusing work in large-scale computations
Popa, L., Budiu, M., Yu, Y., & Isard, M. (2009) · Cited by 89
Self-citation
Cited by
A Survey of Fog Computing: Concepts, Applications and Issues
Yi, S., Li, C., & Li, Q. (2015) · Proceedings of the 2015 Workshop on Mobile Big Data · doi:10.1145/2757384.2757397 · Cited by 1,339
Discretized streams: fault-tolerant streaming computation at scale
Zaharia, M., Das, T., Li, H., Hunter, T., Shenker, S., & Stoica, I. (2013) · Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles · doi:10.1145/2517349.2522737 · Cited by 982
Baccarelli, E., Naranjo, P. G. V., Scarpiniti, M., Shojafar, M., & Abawajy, J. H. (2017) · IEEE Access · doi:10.1109/access.2017.2702013 · Cited by 354
Dias de Assunção, M., da Silva Veith, A., & Buyya, R. (2018) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2017.12.001 · Cited by 304
Big data stream analysis: a systematic literature review
Kolajo, T., Daramola, O., & Adebiyi, A. (2019) · Journal of Big Data · doi:10.1186/s40537-019-0210-7 · Cited by 259
Metadata
- Authors
- Qian, Z., He, Y., Su, C., Wu, Z., Zhu, H., Zhang, T., Zhou, L., Yu, Y., & Zhang, Z.
- Venue
- Proceedings of the 8th ACM European Conference on Computer Systems
- Publisher
- ACM
- Published
- 2013-04-15
- Pages
- 1-14
- OpenAlex
- W1984220394
- Cited by
- 227
- References
- 19
- FWCI
- 27.45
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.