The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
S4: Distributed Stream Computing Platform
Abstract
S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform that allows programmers to easily develop applications for processing continuous unbounded streams of data. Keyed data events are routed with affinity to Processing Elements (PEs), which consume the events and do one or both of the following: (1) emit one or more events which may be consumed by other PEs, (2) publish results. The architecture resembles the Actors model, providing semantics of encapsulation and location transparency, thus allowing applications to be massively concurrent while exposing a simple programming interface to application developers. In this paper, we outline the S4 architecture in detail, describe various applications, including real-life deployments. Our design is primarily driven by large scale applications for data mining and machine learning in a production environment. We show that the S4 design is surprisingly flexible and lends itself to run in large clusters built with commodity hardware.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Simplex Method for Function Minimization
Nelder, J. A., & Mead, R. (1965) · The Computer Journal · doi:10.1093/comjnl/7.4.308 · Cited by 28,905
MapReduce: simplified data processing on large clusters
Dean, J., & Ghemawat, S. (2008) · Communications of the ACM · doi:10.1145/1327452.1327492 · Cited by 18,476
Actors: A Model of Concurrent Computation in Distributed Systems
Agha, G. (1986) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 2,232
ZooKeeper: wait-free coordination for internet-scale systems
Hunt, P. G., Konar, M., Junqueira, F., & Reed, B. C. (2010) · USENIX Annual Technical Conference · Cited by 1,968
Edelman, B., Ostrovsky, M., & Schwarz, M. (2007) · American Economic Review · doi:10.1257/aer.97.1.242 · Cited by 1,567
Aurora: a new model and architecture for data stream management
Abadi, D. J., Carney, D., �etintemel, U., Cherniack, M., Convey, C., Lee, S., Stonebraker, M., Tatbul, N., & Zdonik, S. (2003) · The VLDB Journal The International Journal on Very Large Data Bases · doi:10.1007/s00778-003-0095-z · Cited by 1,495
Actors: A Model of Concurrent Computation in Distributed Systems
Agha, G. (1986) · The MIT Press eBooks · doi:10.7551/mitpress/1086.001.0001 · Cited by 1,401
The Power of Events: An Introduction to Complex Event Processing in Distributed Enterprise Systems
Luckham, D. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-88808-6_2 · Cited by 1,285
The design of the borealis stream processing engine
Abadi, D. J., Ahmad, Y., Bałazińska, M., Cherniack, M., Hwang, J.-H., Lindner, W., Maskey, A. S., Rasin, A., Ryvkina, E., Tatbul, N., et al. (2005) · Cited by 1,255
Condie, T., Conway, N., Alvaro, P., Hellerstein, J. M., Elmeleegy, K., & Sears, R. (2010) · doi:10.5555/1855711.1855732 · Cited by 673
Cited by
Data-intensive applications, challenges, techniques and technologies: A survey on Big Data
Philip Chen, C. L., & Zhang, C.-Y. (2014) · Information Sciences · doi:10.1016/j.ins.2014.01.015 · Cited by 3,047
The rise of big data on cloud computing: Review and open research issues
Hashem, I. A. T., Yaqoob, I., Anuar, N. B., Mokhtar, S., Gani, A., & Ullah Khan, S. (2015) · Information Systems · doi:10.1016/j.is.2014.07.006 · Cited by 2,656
Toward Scalable Systems for Big Data Analytics: A Technology Tutorial
Hu, H., Wen, Y., Chua, T.-S., & Li, X. (2014) · IEEE Access · doi:10.1109/access.2014.2332453 · Cited by 1,099
The role of big data in smart city
Hashem, I. A. T., Chang, V., Anuar, N. B., Adewole, K., Yaqoob, I., Gani, A., Ahmed, E., & Chiroma, H. (2016) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2016.05.002 · Cited by 1,103
Machine learning with big data: challenges and approaches.
L'Heureux, A., Grolinger, K., Elyamany, H. F., & Capretz, M. A. M. (2017) · IEEE Access · doi:10.1109/access.2017.2696365 · Cited by 1,023
Discretized streams: fault-tolerant streaming computation at scale
Zaharia, M., Das, T., Li, H., Hunter, T., Shenker, S., & Stoica, I. (2013) · Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles · doi:10.1145/2517349.2522737 · Cited by 982
Toshniwal, A., Taneja, S., Shukla, A., Ramasamy, K., Patel, J. M., Kulkarni, S., Jackson, J., Gade, K., Fu, M., Donham, J., et al. (2014) · Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data · doi:10.1145/2588555.2595641 · Cited by 956
A survey of machine learning for big data processing
Qiu, J., Wu, Q., Ding, G., Xu, Y., & Feng, S. (2016) · EURASIP Journal on Advances in Signal Processing · doi:10.1186/s13634-016-0355-x · Cited by 894
Bibri, S. E. (2018) · Sustainable Cities and Society · doi:10.1016/j.scs.2017.12.034 · Cited by 799
Mining big data: current status, and forecast to the future
Fan, W., & Bifet, A. (2013) · ACM SIGKDD Explorations Newsletter · doi:10.1145/2481244.2481246 · Cited by 677
Twitter Heron: Stream Processing at Scale
Kulkarni, S., Bhagat, N., Fu, M., Kedigehalli, V., Kellogg, C., Mittal, S., Patel, J. M., Ramasamy, K., & Taneja, S. (2015) · Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data · doi:10.1145/2723372.2742788 · Cited by 570
MillWheel: fault-tolerant stream processing at internet scale
Akidau, T., Balikov, A., Bekiroğlu, K., Chernyak, S., Haberman, J., Lax, R., McVeety, S., Mills, D., Nordstrom, P., & Whittle, S. (2013) · Proceedings of the VLDB Endowment · doi:10.14778/2536222.2536229 · Cited by 536
In-Memory Big Data Management and Processing: A Survey
Zhang, H., Chen, G., Ooi, B. C., Tan, K.-L., & Zhang, M. (2015) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2015.2427795 · Cited by 414
Big data: From beginning to future
Yaqoob, I., Hashem, I. A. T., Gani, A., Mokhtar, S., Ahmed, E., Anuar, N. B., & Vasilakos, A. V. (2016) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2016.07.009 · Cited by 410
Big Data and supply chain management: A review and bibliometric analysis
Mishra, D., Gunasekaran, A., Papadopoulos, T., & Childe, S. J. (2018) · Annals of Operations Research · doi:10.1007/s10479-016-2236-y · Cited by 382
Baccarelli, E., Naranjo, P. G. V., Scarpiniti, M., Shojafar, M., & Abawajy, J. H. (2017) · IEEE Access · doi:10.1109/access.2017.2702013 · Cited by 354
Integrating scale out and fault tolerance in stream processing using operator state management
Castro Fernandez, R., Migliavacca, M., Kalyvianaki, E., & Pietzuch, P. (2013) · Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data · doi:10.1145/2463676.2465282 · Cited by 338
StreamCloud: An Elastic and Scalable Data Streaming System
Gulisano, V., Jimenez-Peris, R., Patino-Martinez, M., Soriente, C., & Valduriez, P. (2012) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2012.24 · Cited by 324
Dias de Assunção, M., da Silva Veith, A., & Buyya, R. (2018) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2017.12.001 · Cited by 304
Benchmarking Streaming Computation Engines: Storm, Flink and Spark Streaming
Chintapalli, S., Dagit, D., Evans, B., Farivar, R., Graves, T., Holderbaugh, M., Liu, Z., Nusbaum, K., Patil, K., Peng, B. J., et al. (2016) · 2016 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW) · doi:10.1109/ipdpsw.2016.138 · Cited by 297
Bibri, S. E. (2019) · Journal of Big Data · doi:10.1186/s40537-019-0182-7 · Cited by 278
Samza: Stateful scalable stream processing at LinkedIn
Noghabi, S. A., Paramasivam, K., Pan, Y., Ramesh, N., Bringhurst, J., Gupta, I., & Campbell, R. H. (2017) · Proceedings of the VLDB Endowment · doi:10.14778/3137765.3137770 · Cited by 276
Towards felicitous decision making: An overview on challenges and trends of Big Data
Wang, H., Xu, Z., Fujita, H., & Liu, S. (2016) · Information Sciences · doi:10.1016/j.ins.2016.07.007 · Cited by 268
Implications of big data analytics in developing healthcare frameworks – A review
Palanisamy, V., & Thirunavukarasu, R. (2019) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2017.12.007 · Cited by 259
Big data stream analysis: a systematic literature review
Kolajo, T., Daramola, O., & Adebiyi, A. (2019) · Journal of Big Data · doi:10.1186/s40537-019-0210-7 · Cited by 259
New Horizons for a Data-Driven Economy
(2016) · Springer eBooks · doi:10.1007/978-3-319-21569-3 · Cited by 253
Adaptive online scheduling in storm
Aniello, L., Baldoni, R., & Querzoni, L. (2013) · Proceedings of the 7th ACM international conference on Distributed event-based systems · doi:10.1145/2488222.2488267 · Cited by 245
Darwish, T. S. J., & Abu Bakar, K. (2018) · IEEE Access · doi:10.1109/access.2018.2815989 · Cited by 232
A survey on the evolution of stream processing systems
Fragkoulis, M., Carbone, P., Kalavri, V., & Katsifodimos, A. (2024) · The VLDB Journal · doi:10.1007/s00778-023-00819-8 · Cited by 207
DSPBench: A suite of benchmark applications for distributed data stream processing systems
Bordin, M. V., Griebler, D., Mencagli, G., Geyer, C. F. R., & Fernandes, L. G. L. (2020) · IEEE Access · doi:10.1109/access.2020.3043948 · Cited by 63
Metadata
- Authors
- Neumeyer, L., Robbins, B., Nair, A., & Kesari, A.
- Venue
- 2010 IEEE International Conference on Data Mining Workshops
- Publisher
- IEEE
- Published
- 2010-12
- Pages
- 170-177
- ISSN
- 2375-9259 (ISSN-L)
- OpenAlex
- W2119745055
- Cited by
- 900
- References
- 12
- FWCI
- 45.68
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.